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Preface 



These notes comprise the contents of lectures I gave at the T.I.F.R. Cen- 
tre in Bangalore in April/May 1983. There are actually two separate 
series of lectures, on controlled stochastic jump processes and nonlin- 
ear filtering respectively, and the corresponding two parts of these notes 
are almost disjoint. They are united however, by the common philoso- 
phy (if that is not too grand a work for it) of treating Markov processes 
by methods of stochastic calculus, and I hope the reader will, at least, 
be convinced of the usefulness of this and of the 'extended generator' 
concept in doing calculations with Markov precesses. 

The first part is aimed at developing optimal control theory for a 
class of Markov processes called piecewise-deterministic (PD proce- 
sses. These were only isolated rather recently but seen general enough 
to include as special cases practically all the non-diffusion continuous 
time processes of applied probability. Optimal control for PD processes 
occupies a curious position just half way between deterministic and Sto- 
chastic optimal control theory in such a way that no standard theory 
from either side is adequate to deal with it. The only applicable theory 
that exists at all is very recent work of D. Vermes based on the gener- 
alized dynamic programming ideas of R.B. Vinter and R.M. Lewis, and 
this is what I have attempted to describe here. Undoubtedly, further de- 
velopment of control theory for PD processes will be a fruitful field of 
enquiry. 

Part II concentrates on the "pathwise" theory of filtering for diffu- 
sion processes and on more sophisticated extensions of it due primarily 
to H. Kunita. The intriguing point here is to see how stochastic partial 



VI 



differential equations can be dealt with by stochastic flow theory through 
what amounts to a "doubly stochastic" version of the FeynmanKac for- 
mula. Using this, Kunita has given an elegant argument to show the 
existence of smooth conditional densities under Hormander-type con- 
ditions. This is included. Ultimately, it rests on results obtained by 
Bismut and others using Malliavin calculus, since one needs a version 
of the Hormander theorem which is valid for continuous (rather than 
C°°) ^-dependence of the coefficients. It was unfortunately impossible 
to go into such questions in the time available. 

I would like to thank Professor K.G. Ramanathan for his kind invita- 
tion to visit Bangalore and K.M. Ramachandran for his heroic efforts at 
keeping up-to-date notes on a rapidly accumulating number of lectures, 
and for preparing the final version of the present text. I would also like 
to thank the students and staff of the T.I.F.R. Centre and of the I.I.Sc. 
Guest House for their friendly hospitality which made my visit such a 
pleasant one. 
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Stochastic Jump Processes 
and Applications 



Chapter 1 



Stochastic Jump Processes 



Introduction 

Stochastic jump processes are processes with piecewise constant paths. 1 
The Poisson process, the processes arising in Inventory problems (stocks 
of items in a store with random ordering and replacement) and queuing 
systems (arrivals at a queue with each customer having random demand 
for service) are examples of stochastic jump processes. Our aim here 
is to develop a theory suitable for studying optimal control of such pro- 
cesses. 

In Section^ martingale theory and stochastic calculus for jump pro- 
cesses are developed. Gnedenko-Kovalenko |16| introduced piecewise- 
linear process. As an example of such a process, consider virtual waiting 
time process (VWT) for queueing systems, where VWT(t) is the time 
customer arriving at time t would have to wait for service, see Fig. (I0.lt . 

Later Davis [7] and Vermes l25ll introduced the concept of piece- 
wise deterministic processes which follow smooth curves (not necessar- 
ily straight lines) between jumps. In Section|2j we will study some ap- 
plications to piecewise-deterministic processes. The idea there is to de- 
rive Markov properties, Dynkin's formula, infinitesimal generators etc., 
using the calculus developed in Section[2 



Service 
Demand 
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Figure 0.1: Arrival time of customers 



1 Martingale Theory for Jump Processes 

Let (X, S ) be a Borel space. 

Definition 1. A jump process is defined by sequences T\, Ti, T$, . . ., 
Z1.Z2.Z3,... of random variables, Tj e R + and 7 T , + i > Tj a.s. and 
Z t e(X,S). Set 

T^ = lim T k . 

k—>oo 

Let zo, Zoo be fixed elements of X. Define the path (x t ) t >o by 

'zo if t<T\ 



x t = < 



Z, if te[T h T M [ 

Zoo II I — -* 00 ♦ 



Then the probability structure on the process is determined by either 
joint distribution for (T t , Z,, i = 1 , 2, . . .) or specifying 

(i) distribution of {Z\, T\) 

(ii) for each k = 1,2, ... , conditional distribution of {S^Z^ \ T^-u i = 
1,2,...), where S t = Tk - Tk-i is then k th inter-arrival time. 

We will start studying the process (x t ) having a single jump, i.e., 
(zo if t<T(co) 



x r 



<<*) 



if t > T(io). 
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If T = oo, let Z = Zoo, a fixed point of X. 



-*-t 



Figure 1.2: 

Define the probability space (D, F, P) as the canonical space for T, 
Z, 

i.e., ((R + xX)f/{(oo, Zoo )},S(R + ) *5,{(oo, Zoo )}, A /) 

where yu is a probability measure on 

((R + X X)f/{(oo, Zco )}, B(R+) * 5, {(oo, Zoo )}). 

The random function (x f ) generates the increasing family of <x- fields 4 
(*?), i.e., 

F? = o-{x s ,s<t}. 



We suppose 



M([0,oo]x{ Zo })f/{0}xX) = 0. 



This assumption guarantees that the process x t does jump at its jump 
time T, i.e., 

P(T >0and Z ± zo) = 1. 

Recall that an R + - valued random variable t is a stopping time of a 
filtration F,, if (t < t e)F t , W. Let 

F t = Completion of F r ° with all F^ - null sets . 

Proposition 1.1. T is not an F® stopping time, but T is an F t stopping 
time. 
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Proof. Let A = {Z = zo) and K be any set in X. Then 

x s \K) = 

\[s, oo] x X)f/([o, *] x {zol) if zo € K and Z(£ - A) n K = <p 

((s, oo] x X)U([o, s] xK) if zo e K and Z(E - A) n K ± (p. 

[o,s]xK ifzot K. 

where £ = R + xX-A. 

Clearly [0, t] x X cannot be in the cr- algebra generated by sets of 
the above form. So T is not an F® stopping time. Let B = X - {zol- By 
assumption, P(A) = 0; so A e F t . 

X ;\B) = [0,t]xX-AeF°. 

So 

[0, t]xXe F t . 

But {T < t) = [0, t] x X. Hence T is an F t stopping time. n 

It can be seen that 

F t = B[o, t] * S UQt, oo]xX)U null sets of F^. 

The stopped cr- field Ft is given by 

F T = {G e Foo : G n (T < t) e F t , V?}. 

Clearly 

Ft = Foo. 

Definition 1.2. A process (M t ) is an F t -martingale if E \ M t \< oo and 
for s < t 

E[M t | F s ] = M s a.s. 

(M t ) is a local F t - martingale if there exists a sequence of stopping 
times S n t °° a.s. such that M" := M t \$ n is a uniformly integrable 
martingale for each n; here tAS n := Min(t,S n ). 
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Proposition 1.2. If M t is a local martingale and S is a stopping time 
such that S >T a.s., then Ms = Mt a.s. 

Proof. Let S n be stopping times such that S n T °° a.s. Then M t ^s„ is u.i. 
martingale. Let M" = M tA s„, Vn. Then by optional sampling theorem 

E[M n s F TASn ] = M n T ; 
but F T = Fee. 



So 



Also 



E[M" \F T ] =M" 



lim M T = M T 



and lim M n s = M s . 

n—>co 

So 

M s = M T a.s. 



Proposition 1.3. Suppose t is an F t -stopping time. Then there exist 
to £ R + such that xAT = to AT a.s. 

Proof. If r is a stopping time, then (tAT < t) e F t , St. But if TAt is 
not constant a.s. on (r < T), then 

(xAr <r)n (]t, oo] x X) c]t, oo[xX for some t e R + . 

But [t, oo] x X is an atom of F t . This contradicts the fact that r is a 
stopping time. So 

tAT = t ATa.s. 

The general definition of a stopped cr-field is that if U is a stopping 
time. Then 

F v = {A € F | A n (U < t) e F t , Vt). 



But this is an implicit definition of the cr-field. 
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Exercise 1.1. Suppose r = to AT. Show that 
(i) F T = F to 
(ii) F T = ct{x tAs , S > 0}. 

Definition 1.3. For A e S, define 

F A (t) = p([t, oo] x A) 
and F(t) = F x (t) = P[T > t]. 

Note that F(0) = 1 and F(.) is monotone decreasing and right con- 
tinuous. Define 

_ Jinf {t : F{t) = 0} 
C " \+oo if [t : F(t) = 0} = <p. 

Proposition 1.4. Suppose (M t ) t >o is an F t local martingale. Then 

(a) if c = oo or c < oo and F(c-) = 0, then M, is a martingale on [0, c[. 

(b) ifc < oo, F(c-) > 0, then (M t ) is a uniformly integrable martingale. 
Here F(c-) = lim F(t). 

Proof, (a) If Tic > Ta.s. for some k, then 

M tKTk = M tATkAT = M tAT = M t . 

So M t is a u.i. martingale. Hence suppose PIt^ < T] > for all 
k(*); then by Proposition 1 1.31 

T^AT = tfrAT for some fixed % 

and tjt < c because of (*). Also tk T c since tj T °°- 

So 

M t An = M tATkAT = M tATAtk = M tAtk . 

Hence M tAtk is a u.i. martingale. So (M t ) t<c is a martingale. 
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(b) c<oo,F(c_) >0. 



F(c-) = P(T = c) 



P(T = c) > 0; so it must be the case that tk = c for some k. Oth- 
erwise P(Xk < c) > F(c-) > 0; so "tfc T °° «••?•" fails. For this 

M tAtk = M, 

So (M t ) is a w.j. martingale. 

Our main objective is to show that all local martingales can be rep- 
resented in the form of "stochastic integrals". So we introduce some 
"elementary martingales" associated with the process (x t ). For A e S 
and t € R.+, define 



p(t,A) = / ( j> r) 7 (ZeA) 
p{t, A) 



J F(s.) 



dF A {s). 



]o,TA t[ 



Proposition 1.5. Let q(t,A) = p(t,A)-p(t,A). Then (q(t,A)) t >o is an F t 
- martingale, i.e., p(t,A) the "compensator" of the point process p(t,A). 

Proof. (Direct calculation). Take t > s, then 

F A (s) - F A (t) 



and 



E[p(t,A)-p(s,A)\F s ]=I is<Ty 
E[p(t,A)-p(s,A)\F s ]=I s<T 
F(t) 



F(s) 



F(s) 



J F(u-) F(s) J J F(u-) 



[s.t] 



[s.t] [s.r] 



I J§^^-IfhI mr)dFAM 



[s,t] [s,r] 



[s,t] 



[u,t] 
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J F(u- 

[s.t] 

r dF A (u) A , 

F(t) — -^ +F A (t)-F / 
J F(u-) 



-(F{t)-F(u-))dF A (u) 

I s.l I 



(5). 

s.t] 



So 



£[?(/, A) - ? (5,A) | F s ] = 



Another expression for p(t, A): We have F A (.) « F(.) (i.e. F A (.) is 
absolutely continuous w.r.t. F (.))• So there exists a function ^(s, A) such 
that 

F A (0) - F A (0 = - \ A(s,A)dF(s). 



]0.t] 

In fact 

A(s,A) = P(ZeA\T = s). 

Suppose X is such that a regular version of this conditional probabil- 

-dF A (s) 

ity exists (which is the case, since X is Borel space). Then = 

F(s-) 

-dF(s) 

A(s,A)dA(s) where dA(s) . 

F(s-) 

Then 

p(t, A) = I A(s,A)dA(s). 

]o,T A T] 

10 Stochastic Integrals 

Let / denote the set of measurable functions g : D. — > R. such that 

g(°°,Zco) =0. 

(a) Integrals w.r.t. p(t, A): Suppose N t is a counting process. Since 
its sample functions are monotone increasing and there is a one- 
to-one correspondence between monotone increasing functions 
and measures, and since in this case, mass is concentrated at the 
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jump points and they are only countable; the function N t defines 
a random measure on (R, S(R)) say, n = £ ^r, where 6x is the 

i 

Dirac measure at x. Similarly, the one jump process can be iden- 
tified with the random measure 8(t,x t ) on R+ x X. So we can 
define Stieltjes integrals of the form J g(t, x)p(dt, dx) for suitable 
integrands g e I as 

I g(t, x) p(dt, dx) = g(T, x T ). 

We say g e L\(p) if 

E I \g(t, x) | p(dt, dx) < oo 
n 
and denote 

|| g || Ll(p )= £ I | g(t, x) | p(dt, dx) 

.Q 

Clearly g e L\(p) if and only if 

I | g(t, x)dp < <x> 

R + xX 

(b) Integrals w.r.t. p(t, A): ll 

Recall p(t, A) = J A(s,A)dA(s). 

]o,T A t] 

So we define 

I g(t,x)p(dt,dx) = I I g(t,x)A(t,dx)dA(t) 

Q. [o,T] X 

and say g e L\{p) if I |g(/, x) | p(dt,dx) < cxs 

n 

and || g \\ Ll (p)= I I g(t,x) | p(dt,dx). 

.Q 
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Proposition 1.6. 

and so 



llLi(p)-H 8 llLi(p) 



Proof. 



Li(p) = L x {p). 
Wuip) = - I I ^-— I gO,x) I dp(s,x)df(T) 

R+ [o,T] 

= /7V73 I S(s,x) I (- J dF(t)W(s,x) 

a [.v,oo] 

= I |gO,x) \dp(s,x) 
n 

= 11 5 llLi(p) ■ 



Define 



L l ° c (p) = {g€l\ g(s, x)I s < t e Uip), W < c] 
L l ° c (p) = {gel\ g(s, x)I s < t e U(p), W < c). Clearly 
Lf (p) = Lf (p). 

12 Following is the main result of this section, which gives an integral 

representation for F t local martingales. □ 

Proposition 1.7. All F r local martingales are of the form 

M t = I (g(s,x)I (s < t) dq(s,x) 
n 

= I (gO, x)I s <,dp(s, x) - I (g(s,x)I (s < t) dp(s,x). 



n si 

loc/ 



/or some g € Lf c (p). 

We need the following result. 
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Lemma 1.1. Suppose (M t ) t> o is u.i.F t martingale with Mq = 0. Then 
there exists a function h : Q, — > R ^wc/z ^/jfitf 

£|/?(r,z)|<oo (i) 

and 

M t = I if > T) h(T,Z)-I (f<T) j- f h{s,z)p{ds,dz) (2) 

]0,t]xX 

Proof. If (M f ) is a tu. martingale, then M t = E[g \ F t ] for some im- 
measurable r.v. £ and from the definition of F M , we have 

£ = h(T,Z) a.s. 

for some measurable h : £2 — > R. Expression Q is satisfied since M, is 13 
u.L, and M = implies 

fh.dp = 0. (3) 

n 
Now 

M, = £[/i(r,Z)|F,] 

= I(t>T)h(T,Z) + I it<T) -— I h(s,x)p(ds,dx). (4) 

From (J3jl and ©J we have (J2ji. n 

For g e L}j OC (p), the stochastic integral 



Mf = I g(s,x)q(ds,dx) 



]0,i]xX 
is defined by 



f f = I I( s <t)g{s,x)p{ds,dx)- I I {s < t)g( ^ x) p{ds,dx). 



Mf 

<-xX R+xZ 



Then the question is whether M t given by (J2ji is equal to Mf for 
some g. As a motivation to the answer consider the following example. 
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Example 1.1. Let (X,S) = (R,B(R)) and 

/u(ds, dx) = if/(s, x) ds dx. 
Then 



Mt = I 



(t>T) 



g(T,Z) 



I I f7~^(s,x)t]j{s,x)dxds 



l{t>T) 



J7^> s(s ' 



x)ifr(s, x)dxds > (5) 



o 



14 If My given by (J5J) is equal to M t given by ©, then the coefficients 

of I(t>T) and /(r<r) must agree. Comparing the coefficients of I( t >T), we 
require 

I 



(.v) 



g(s, x)i}/{s, x)dxds. 



Let 



Define 



and 



Then 



r&) 



T](t) = h(t,z)-g(t,z). 

y(t) = I iphdx 

f(f)= fifrdx. 

R 
VR 

J F(s) J F(s) 





i//(s,x)dx)ds 
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that is 

d f(f) 1 

j t ^-m n{t)+ W) yit) 

T](0) = 

which has a unique solution 

t 

n(t) = I cp(t,s)——y(s)ds, 
J F(s) 



where 

f /(») 

0(r, 5) = exp t^-t«« 

F(5) . dF(/) 

= , since rm = . 

F(0 Jr 

So 15 

f 

»7(0 = J7~: J y(s)ds. 
o 
Hence 

g(t, z) = h(t, z) h I I h(s, x)ilf(s, x)dxds (6) 

F(0 J J 

R 

Now it can be checked that with this choice of g the coefficients of I(t<T) 
in © coincides with that of ©. So M t = Mf. 

Now we can prove the general case given in Proposition ll.7l 

Proof of Proposition 1.7. 

Case 1. c < oo,F(c_) > 0. Take a local martingale M t with M = 0. 
Then (M t ) is u.i. So M, = E[h(T,Z)\F t ] for some measurable h such that 
E\h\ < oo, Eh = 0. Then we claim that M t = Mf where 

g(t,z) - h(t,z) + I(t<c)-£7T I h(s,z)/u(ds, dz). 

]°,i]xx 
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But this can be seen algebraically following similar calculations as 
of the example ITTTI Now to show that g e L l ° c (p). 

f\g\dp < jwdp - r J- c \h\dpdF( t ) 

]o,c[ ]o,(]xX 

<f\h\dfi-— l — J J \h\dpdF(t) 

]o,c[]o,f]xX 

< J \h\dp + — l — J (F(t) - F(c.))\h\dfi 



ib)/' 



]o,c[xX 



16 Case 2. c = oo, or c < oo and F(c_) = 0. Then from vrovosition \l.4\ M t 
is a martingale on [0, c], and so it is u.i. on [o, t]for t < c. Therefore 

M s = EihiZZ)^] 

for some function h satisfying 

I \g(s, x)\dp(s, x) < co for all t < c. 

]o,t]xX 

Define g(s,x) as in ©. Then calculations as in case\I\ Show that 
M s = M 8 s for s < t < c. Now 



f \g\dp < f \h\dp - f -j±-< f \h\dp dF(i 

>,t]xX ]o,txX ]o,i] ]o,s]xX 

< J \h\dn(l - f ^-dF(s)) 



]o,t]xX ]o,t] 

< oo for t < c. 



Hence g e L" g (p). 
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Conversely, suppose g e L° 8 (p). Then it can be checked that Mf is 
a local martingale. 

Remark 1.1. If g e L" 8 (p) then Mf is a martingale. But the result does 
not say M t is a martingale if and only if M = M 8 for g e L\{p), it only 
characterizes local martingales. 

Remark 1.2. All preceding results hold if z is a random variable; then 
p should be taken as conditional distribution of (T, Z) given z 

The multi-jump case: The process x t has jump times T\,T2, ■ ■ ■ with 17 
corresponding states Z\,7ji, . . . Let (Y,y) denote the measurable space 

(Y,y) = ((R + X X)t/{(oo, Zoo )}, o-{B(R + ) * S, {(«,, Zm )}}). 

Define 



"-"In*] 



*'=1 !=1 



where (7,-, y/) denote a copy of (T,y). Let 

S k (u>) = T k (u>) - T k -i(io) 
and w k {co) = (S(a>),Zi(co), . . .S k (a>),Z k (a>)). 



Then 



7*(w) = £$,•(") 

i=\ 

L(w) = lim r^(a>). 

k—>oo 

As before, (x ? (w)) ? >„ is defined by 

'zo if f < T(a>) 
x l Uo) = \z k if te[T k (oj),T k+1 (oj)] 
Zoo if t > T^ico) 
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A probability measure p on (Q, F°) is specified by a family p' : 
fij-i X y — » [0, 1] (with Q = <p) satisfying 

(i) //(.; T) is measurable for each fixed T 

(ii) /j.'(wj-i(oj); .) is a probability measure on (7,;y) for each fixed <x> e 

a 

18 (iii) //Wi(w); .({0} x X) U (R+ x fo_i(a>)})) = for all o), 

(iv) fi l (Wi-i(a>)\ .{(oo,Zoo))) = 1 if5j-i(w) = oo. 
Then for T e y and rj e Q ( _i,/i is defined by 

//[(r 1 ,z 1 )er]= / / 1 (r) 

/i[(S/,Z0 e r|w/_i = 17] = fa : T)j = 2,3 • ■ • 

Notice that as in the single jump case, here (iii) ensure that two 
"jump times" 7Vi, T do not occur at one and that the process x t does 
effectively jump at its jump times (iv) ensures that//[Z,t = z<»|7* = 00] = 
1. 

As before, F" = cr{x s , s < t\ and 

F t = completion of F° with all //-null sets of F°. 

Proposition 1.8. (i) F^ = F, the completion of F° 

n 00 

(H) Ft,, = II Ji X n W- 

i-l i'=n+l 

The idea here is to reduce everything to one jump case. That is, the 
process "restarts" at each 7\. We need the following result. 

Proposition 1.9. 

F {Tk _ l+t) AT k = F Tk _ l V(r{x(y k _ l+S )KTk,o < s < f)} . 

Proof of this is an application of the "Galmarino test" (Dellacherie 
and Mayer [12], theorem IV, pp. 100). 

19 Recall that in one jump case F to xr = Ft,, ■ Now we conjecture that, 
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19 



if U = (7jt_i + t )AT k , then 



(k-\ \ 


t 
* jt * 


f CO \ 
J=k+1 y 



where y£ = S * B[o, t (> ] U(ix [t , oo]) . 

As an example, see the following exercise. 

Exercise 1.2. Consider a point process with k = 2, and take the proba- 
bility space as R+. Then 

2 

Then 

(a) Show that 

F t = Borel sets in {S i + S 2 < t) 

+ (A x R + ) n 5 + [f, oo] x R + , A G B(R) 

where 

5 = {5i+5 2 >f}P|{5 2 <f}. 

(b) With £/ = (Tx + ?„)Ar 2 , show that 

F v = B(R + x [o,t]) + {(A x R + ) n(R + x]f , ex,]) : A e B(R)}. 

Elementary Martingales, Compensator's Define 
pa,A) = 2 7 (f>ro 7 fc6A) 

which counts the jump of (x t ) ending in the set A. Define 

J F\u) 

[o,s] 



<Pi(s) 



-dF 1A (u) 
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where 20 

and 



F 1 =/i'([l,co]xI) 



- f— 



4(w k -us)= ■ I -—dF^iu) 



where 



Now define 



[«,.v] 



F M W=//Wi;[«, M ]xA). 



p{t, A) = 0?(rO + ^(wi;S 2 ) + • ..^(wy_i;f- r/_i(w)) 
for t€\Tj. lt Tj\. 

Exercise 1.3. Consider a renewal process 

X t = 2j r (t>Ti) 
i 

and S j 's are independent, P(S ,■ > t) = F(/) is continuous. Then show 
that the compensator for x t is 

p(t) = -(niFiS l )FiS 2 )---FiS k - l )Fit-T k - l ))forte[T k - X ,T k \, 
and x t — pit) is a martingale. 

Example 1. If Fit) = e~ at , then pit) = at. 
21 Proposition 1.10. For fixed k, and A e S, 

qitAT k , A) = pitAT k , A) - pitAT k , A),t>0 
is an F t - martingale. 
Proof. Calculation as in proposition II .51 □ 

The class of integrands / consists of measurable function git, x, a>) 
such that 



git, x, 0)) 



g\t,x), t<Txia>) 

g k iwk-i,t - T k -i, x), t e\T k -iioS), T k io))}, 
0, t > Tovioj) 
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for some function g k such that g l (oo, x) = g k {w k ', oo, x) = 0. Such g's 
are F\ -predictable processes. Now we define L\(p),L\{p), etc. exactly 
as in one jump case: 

fgdp = Y 4 g(T i ,Z i ) 

i 
L i (p) = \g£l:EY J \g(T i ,Z i )\<J> 

I gdp = - ^ I g{m-u s, x)A{o) k ^, s, dx) 



dF K (s) 
'F k (s-) 

k ]o,T k -T k _,]xX 

where 

dF kA 
A(u) k - h s,A) = -77^0) 

g e T l ° c (p) if there exists a sequence of stopping times cr k 1 T^ a.s. and 
gl( t <cr„ e L\{p), Vn. For g e L\ oc (p) we define 






Mf = J g(s,x)q(ds,dx) 
[o,t]xX 



= I g(s,x)p(ds,dx) - I g(s, x) p(ds,dx). 

[o,t]xX [o,f]xX 

Proposition 1.11. If g e Lj oc then there exists a sequence of stopping 22 
times T n < T^ such that r„ | 7"oo and Mf. T is a u. i. martingale for 
each n. 

Proof. Take r n = nAT n Acr n . Then the result follows by direct calcula- 
tions using the optional sampling theorem. □ 

Now let (M t ) t >o be a u.i. F,-martingale. Then 

ex? 

M t = M tATl + Yj( M tm - Mt^), / r > Ti _, (7) 

k=2 

because this is an identity if t < r M and the right-hand side is equal to 
lira Mt k is t > r M . Here we have Mj m - = Mj^. Now we state the main 
result. 
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Theorem 1.1. Let (M t ) be a local martingale of F t . Then there exists 
g e l}° c {p) such that 

M t - M Q = I g(s, x, to) q(ds, dx). 
[o,t]xx 

Proof. Suppose first that M t in a u.i. martingale, define 

x] = M tATl 

A - M (t+Tt _ l)ATk - M Tk _ x ,k = 2,3,... 



Then from © 

A/I — \ . 

\t-T k .,)Vo- 



M t = ^A 



k=i 

We can now use proposition 11.71 to represent each x k . Fix k and 
define for t > 0. 

H, = F {t+Tk _ i)ATk . 

23 Then x) is an H t martingale. Then there exists a measurable function 

h k such that 

x k t =E(h k (co k ^;S k ,Z k )\H t ). 



Then using proposition [L7] there exists g (co k -i; s,z) such that 

x k = I g k (u) k -\;s,z)q k (ds,dz) 
]o,/]xZ 

where q k {t,A) = q((t + T k .{)AT k ,A) and g k e L l ° c (p k ) for all c/ -1 
a.s. Piecing these results together for k = 1, 2, 3, . . . gives the desired 
representation with g = (g k ). It remains to prove that g e L l ° c (p) as 
defined, for which we refer to Davis |6|. 

If (M t ) is only a local martingale with associated stopping time se- 
quence t„ T oo such that M t Ar n is a u.i. martingale, apply the above 
arguments to M r Ar„ to complete the proof. □ 

Corollary 1.1. If T^ = oo a.s. then the result says (M t ) is a local 
martingale ofF t if and only ifM, = Mf for some g £ L l ° c (p). 
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Remark 1.3. It would be useful to determine the exact class of inte- 
grands g required to represent u.i. martingales (as opposed to local mar- 
tingales) when the jump times Tj are totally inaccessible, Boel, Varaiya 
and Wong [4] show that {M g ,g e L\(p)) coincides with the set of u.i. 24 
martingales of integrable variation. It seems likely that this coincides 
with the set u.i. martingales if Ep(t, E) < oo for all t (a somewhat 
stronger condition than r ; ^ oo a.s.) but no proof of this is available as 
yet. 

2 Some Discontinuous Markov Processes 

Extended Generator of a Markov Process 

Let the process x t e (E, E), some measurable space. Then (x t , F t ) is 
a Markov process if for s < t 

E[f(x t )\F s ] = E[f(x,)\x s ]a.s. 

A transition function p(s, x, t, F) is a function such that 

p(s,x s ,t,T) = P{x t e T\x s ) 

= E[Ir(x t )\x s ] a.s. for t > s. 

p satisfies the Chapman-Kolmogorov equation 

p(s, x, t,T) = I p(s, x, u, dy)p(u, y, t, T) for s < u < t. 

E 

Not every Markov process has a transition function, but usually one 
wants to start with transition function and construct the corresponding 
process. This is possible if (£, E) is a Borel space (required to apply 
Kolmogorov extension theorem; refer Wentzel |27|). One constructs a 
Markov family, 

{P x>s ,(x,s)eExR + }p XtS 

being the measure for the process starting at x s = x. All measures P sx 
have the same transition function p. Denote by E x>s integration w.r.t 
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P xs . Let B(E) be the set of bounded measurable functions 
/ : E -» R with ll/H = sup |/(x)| 

25 Define 

r ijr is an operator on B(E) such that 
(i) it is contraction, \\T S J\\ < \\f\\,T s ,,\ = 1. 
(ii) Semi group property: r < s < t, 

' r,t = ' r,s* s,t 

for 

T r ,t(T s ,,f)(x) = E x , r [E Xs , s (f(x t ))] 
= E x , r [E(f(x t )\F s )] 

= Ex,rf(Xt)) 
= T r J{x). 

T SJ is time invariant if T s+rJ+r = T Stt for all r > —s. Then T Stt = 
Tqjs = T t - S . So T is a one parameter family; this happens when the 
transition function is time invariant i.e., p(s, x, t, T) = p(s + r,x,t + 
r, T). Then get a one parameter family of measures (P x , x e E) and the 
connection is 

T t f(x) = E x f(x t );T Q f = f. 

Let 

B (E) = {fe B(E) : \\TJ - f\\ -» 0, U 0}. 

o o 

An operator A with domain D(A) c B (E) is the strong infinitesimal 
generator of T t if 



lim||(r,/-/)-A/||=0. 

fj,0 



26 So 
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Af = -T t f(x)\ t=Q 
at 



Take / e D(A). Then 

lim -{T t TJ - TJ) = lim -(T t+S f - TJ) 
40 r 40 f 

= iimr,i(r,/-/) 

40 t 
= T s Af. 

So / € £>(!) implies T,/ e D(A) and ir,/ = T,!/. So we get 
backward Kolmogorov equation 

^-TJ = A(TJ). (1) 

as 

The main results of the "analytic theory" of Markov semigroups are the 
following; 

(i) Hille-Yosida theorem: Necessary and sufficient conditions for an 

o 

operator A to be the generator of some semigroup. 

o o 

(ii) If A satisfies these conditions, then D(A) is dense in Bq(E) and 

o o 

(A, D(A)) determines T t (via the so called resolvent operator). 

o 

NB: The domain DA provides essential information. 
Integrating Q, we get Dynkin's formula 

t 

T t f(x)-f(x) = JT s Af(x)ds 

o 
t 

i.e., E x f(x t ) - f(x ) = E x Ja/(x s )cIs, f e £>(A). 

o 



26 



1. Stochastic Jump Processes 



Proposition 2.1. Iff e 0(A) then the process 



C{ = fix,) - f(x ) - I Af(x,,)ds 






is a martingale. 
27 Proof. For t > s 



E\C{ - C f s \F s ] = E 



f(x t ) - f(x s ) - f Af(x u )du\F s 

s 

t 

E.JM-fM-E.Jifl,,*,, 0. o 



Definition 2.1. Let M(E) be the set of measurable functions f : E — > R. 
Then A,D(A) with D(A) c M(E) is the extended generator of(x t ) ifc{ 
is a local martingale, where 



C{ = f(x t ) - f(x ) 



I 

- JAf(x s 



;)dS. 



o 



This is an extension of {A, DA)) in that D{A) c D{A) and Af = Af 

o 

for f G !D(A). We have uniqueness of A in the following sense. Write 

t 



fix,) = f(x ) + JAf(x s )ds + C{. 



This shows that (f(x t )) is a "special semi-martingale " (=local mar- 
tingale + predictable bounded variation process). The decomposition is 
unique. So, if B is another generator then 
t 

i (Af(x s ) - Bf(x s ))ds = 0P x a.s.\/t. 
o 
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Thus Af(x) = Bf(x) except on a set of potential zero, where a set of 
r has potential zero, where a set T has potential zero if 



oo 



E x Ir(x s )ds = Vx. 

o 

Example 2.1. Suppose x, e M. d satisfies 

dx t = b(x t )dt + cr(x f ) dw t 
with standard Ito conditions. If / e C 2 , then 
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df(x t ) 



Y^ df l r , #/ 

1. J •> 



dxj 






dt + I V f'crdw. 

o 



So C 2 c D(A) and 



t 



NB: This is not a characterisation of 0(A). 

Remark 2.1. If we had required C\ to be a martingale rather than a 
local martingale in definition 12.11 then not every f e C 2 would be in 
D{A) because of the properties of I to integrals. 

Exercise 2.1. For i = 1,2,... Jet N' t be a Poisson process with rate Aj 
where X,/i; < oo. Define 

oo 
i=\ 

where l{ > and 

CO 

J] liXi = r<oo. 
i=\ 
Find the extended generator of x t 

(This is also an example where jump times are not isolated). 
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Piecewise-linear Markov Process 29 

Gnedenko-Kovalenko introduced the concept of piecewise linear 
Markov process. Later, Vermes [24] simplified the definition as fol- 
lows. A piecewise linear process is a two component Markov process 
(x t ) = (v t , % t ) where v t is integer- valued and g t takes values in an interval 
\a n , b n ] of the real line if y t = n (b n may be +oo). Let E be the state 
space, i.e., E = {(n, £) e Z x R : £, e Z x R : £ e [a n , b n ]} Then the prob- 
abilistic description is that if the motion starts at (n, z) e E and x t os 
given by v t = n,^ t = z + t for t < T\, the first jump time. " Spontaneous 
jumps" happen at rate A(x t ), i.e., probability "jump occurs" in (t, t + dt), 
is A{x t )dt, and process must jump if % t - = b n . Let the transition measure 
be given by Q(A; x) for A e B(E). Then jcr, is selected from the prob- 
ability distribution <2(A;x7-,). After a jump, motion restarts as before. 
Thus the law of the process is determined by specifying the intervals 
[a n ,b n ], the jump intensity A(x) and the transition measure Q(A;x). 

Example 2.2. Non-stationary countable state Markov Process (£ r ) 

(£ T ) takes integer values with the-dependent transition rates atj(t) 
such that 

PT% t+h = i|£ = ;] = aij (t)6 + o(6), i * j. 

Then x t = (£ f , i) is a PL process with no barriers, i.e., a n = 0, b n = +oo. 

30 Example 2.3. Countable state process with non-exponential sojourn 
times 

Here, jump times of the process (x t ) form a renewal process with 
inter arrival density b{.) and transition matrix q t j = P[xj k , = i, xj k = j]. 
This is a PL process with v, = x t , and g t the time since last arrival. The 
jump rate is 

b(0 



My,& 



J b(t)dt 
i 



Here again a n = 0, b n = +oo. 

Example 2.4. Virtual waiting times. (The M/ G/ 1 queue) 
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Customers arrive at a single-server queue according to Poisson pro- 
cess with rate //, and have i.i.d. service time requirements with distri- 
butions F. The virtual waiting time % t is the time a customer arriving at 
time t would have to wait for service. Piecewise linear process structure 
is : 



II if queue is not empty 
if queue is empty 



and a\ = 0, b\ = oo,a = b = 0. Here £ r moves to left with uniform 
speed and transition to (0, 0) is certain if x r _ = (1,0). 

A more general definition of PL process of Gnedenko and Kova- 
lenko allows £j t to move in an open subset V , of M. d (v t ) with uniform 
speed in a fixed direction V(y t ). Again transition must take place if 
£ r _, € d0 Vt , the boundary of V( . 

Example 2.5. VWT with renewal process arrivals. (The GI/G/I queue) 31 

Suppose the inter arrivals times in Example l2.4l are not exponential, 
but form a renewal process with inter arrival density b{.). Now the ap- 
propriate structure is v is or 1 as before, d{\) = 2, d(0) = 1. (When 
v = 1 we have to remember both the value of VMT and the time since 
the last arrival.) 

We cannot accommodate this in previous framework, because there 
[a n , b n ] is fixed, whereas here the length of the interval is random. 

Davis [7 1 introduced the piecewise deterministic (PD) process 
which is a further generalization. It is similar to the piecewise linear pro- 
cess, except that £ t satisfies some ordinary differential equation, rather 
than moving in straight line. 

Example 2.6. Shot noise. This has sample functions similar to the VWT 
process except that decay between arrivals is exponential rather than 
linear (fig. O. 
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Figure 2.1: 

32 Example 2.7. A model for capacity expansion. Suppose that the demand 
for some utility is monotone increasing and follows a Poisson process 
with rate p. Each 'unit' of supply provides q units of capacity. These 
are built one at a time at a cost of Rs.p. Investment takes place at a rate 
of Rs.«(0/week and u(t) < constant. When 



I 



u(s)ds = p, 



then the project is finished, capacity is increased by q and investments 
are channelled into next project. 



Denote d t = demand; c t = capacity at time t; i; t = cumulative invest- 
ment in current project 



; 



u(s)ds, 



where t is the last time project was completed. Investment is determined 
by some "policy" i/f, i.e., 

d 
u(t) = —J, = i/f(ct, d t , 6) 
dt 

where (c t , d t , % t ) is the current "situation". Define v t = (c t , d). Then the 
process x t = (v t ,^ t ) evolves in the state space E = Z\ x [o, p] {Z\ 
is the 2-dimensional positive integer lattice). Then for v = (c,d) if 

d 
gv(g) = Hc,d,0^t satisfies — & = g v ,(&)- 
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The piece wise-deterministic process; 

Let K be a countable set and d : K — > N (= natural numbers) be a 
given function. For each ve^, M v is an open subset of R d( - V \M V can be 
a (i(v)-dimensional manifold). Then the state space of the PD process is 33 

E = U M v = {(v,£);v € £,£ e M v }. 



Let 



E = [u'A v ;A v eB(My)} 



Then (E, E) is a Borel space. Then the process is x t = (v t , g t ). The 
probability law of (x t ) is specified by The probability law of (x t ) is spec- 
ified by 

(i) Vector fields (X v , v e K) 

(ii) A 'rate' function X : E — > R + 

(hi) A transition measure <2 : £ X E — > [0, 1] 

Assume that corresponding to each X v there is a unique integral 
curve <p v (t,z), i.e., <p v (t,z) satisfies 

at 

<f>y{0,Z) = Z 

for every smooth function /, and <p(t, z) exists for all t > o. Let dM v 
be the boundary of M v . d*M v is those points in M v at which integral 
curves exit from M v , i.e., d*M v = {z € dM v : (f> v (t,^) = z for some 

y£RxM v ). 
Let 

T* = {v,z:v:eK, z ed*M v }. 

So T* is the set of points on the boundary at which jumps may take 
place. For x = (v,z) e E, denote 

**(*) = inf{^ > : <f>y(t,z) e d*M v ). 



32 1. Stochastic Jump Processes 

Write Xh(x) for the function whose value at x = (v, z) is X v h(v, ,)(z). 
For A, we suppose that the function t — > A(y, <p v (t, z)) is Lebesgue inte- 
grate on [o, e] for some e> 0, 0(., x) is a probability measure on (E, E) 34 
for each x e EUT*. 

The motion of the process (x t ) starting from x = (n, z) £ E is de- 
scribed as follows. Define 



F(t) = 



e\p\- J A(n,cf> n (s,z))ds\, t < t*(x) 
0, t > t*(x). 



This is the distributions of T\, the first jump time. More precisely, 
F(t) is the survivor function 

F(t) = P x [Ti > t]. 

Now let Z\ be an £-valued random variable with distribution 

Q(, ; cp n ){Ti , z)). Then define 

Un,4> n (t,z)) t<T x 
x t = < 

{ Zi t = T l 

and restart with (n,z) replaced by Z\. Assume 7^ | oo a.s. Then x t de- 
fines a measurable mapping from (£1, a, P) (countable product of unit 
interval probability spaces) into space of right continuous E- valued 
functions. This defines a measure P x on the Canonical space. 

NB: The condition on A ensures that T\ > a.s and hence that 7\ - 
7Vi > a.s. 

Proposition 2.2. (x t , P x ) is a Markov process. 

Proof. Suppose that T x < t < T^+i- The distributions of 7\+i - Tj, is 
given by 



P[T k+l -T k >s] = { 



exp \- § A{v Tk ,<p Tk )du \, s<f(x Tk ) 
0, s>t*(x Tk ). 
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35 Denote v = vj k ,^ = £,j k , Then for s > t and s < t*{xT k ) 

P[T k+l > s\T k , T M >t] = P[T k+l -T k >s- T k \T k , T M - T k > t - T k ] 

s-T k 



exp 



exp 



- I A(v, <p(u, i;))du 



t~T k 
s-t 

-I 



My,, (p v .(u, %t))du 



where we used the semigroup property of (p. Since the process "restarts" 
at 7\ + i, the law of the process for s > t given part upto t coincides with 
the law given x t . Hence the Markov Property. □ 

Let T c T* be the subset for which, if y = (v, £) e T* 

P[T\ = T*(x)] -> asx = (n,z) -> y. 
Then T is called the "essential" boundary 

Exercise 2.2. Prove that y e T if and only if P X \T\ = T*(x)] > for 
some x = (v, z). 

So P x [x s - e T - T*] for some s > 0] = 0, and Q(A, x) need not be 
specified for x e T - T*. 

Example 2.8. Here v has only a single value; so delete it. £ takes values 

inM = [0.1] xR+,/1 = OandX = . Then T* = \{l,y);y e M.+ }. Let 

Q{.,{l,y)) = 6{\- l -y, l -y). 



Then starting at xq = (0, 1) we have 

n ^ 

T n = ) so that lim T n < oo. 
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k=\ 



The same effect could be achieved by the combined effect of A and 
Q, suitably chosen. So, we prefer to assume r M = oo a.s. rather than 
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stating sufficient conditions on x, A, Q to ensure this. To illustrate the 

difference between T and T*, suppose that M£\, 1) = (This is 

equivalent to saying T\ is uniformly distributed on [0, 1] if the process 
starts at (0, 1). Then (1,1) os never hit, whatever be the starting point. 

So(i,i)er-r*. 




The Associated Jump Process. 

Let (x f ) be an PD process. Definite the associated jump process (z t ) 
by 

z t = x Tk ,te[T k ,T k+l [ (2) 

37 This is an £-valued jump process such that Zj K = xj K . Let F t = 

<t{x s , s <t) and F) = o-{z s , s < t). 

Proposition 2.3. F t = F\ for each, t 

Proof. This follows form the fact that there is a one-to-one mapping 
from X[ 0tt ] to Z[ ,t]-x — » z is given by ©. Conversely, if Z[ ,t] is given 
then x[o, t] can be constructed since the motion in the interval [T k , T k +\ [ 
is deterministic. □ 



NB: 



(1) x t and z t are not in one-to-one correspondence at each fixed time t. 
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(2) (z t ) is not a Markov process. 

Since F t = F], we can apply jump process theory. Define 

Tt<t 

P*t = ^jhxTt-eT) 
Ti<t 
t t 

p{t, A) = f Q(A, x s )A(x s )ds + f Q(A, x s _)dp* s (3) 



Proposition 2.4. Suppose E(p(t,E)) < oo. Then for each 

AeE,q(t,A) = p(t,A)-p(t,A) (4) 

is an F t -martingale. 
Proof. From previous results, the compensator of p(tAT\ , A) is 

]o,tATi] 

But 38 



F t 



exp - J A(x s )ds t < t* (x) 

t > t\(x). 



dFt_ 
F, 

af, 



Thus --^- = A(x t )dt for t < t\(x) and 



1. 



Ff- 



This verifies the result for t < T\ . As before, we show by consider- 
ing intervals [7V i, 7\] that the compensator of p(tATn, A) is p(tAT n ,A) 
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given by (J3J). Since p(t,A) and p(t,A) are monotonic increasing func- 
tions and T„ 1 oo a.s. E(p(t, E)) < oo, taking the limits, we have 

q{t,A) = p(t,A) -pit, A) 

is a martingale. 

Exercise 2.3. Show that p* is an F t — predictable process. 

Then (|4]i is the Doob-Meyer decomposition of the submartingale p. 

The next step is to use stochastic integrals to calculate the extended 
generator of x t . Choose the following integrands. For Measurable / : 
E -> R, define 

Bf(x,s,w) = f(x)-f(x s -(co)) 

ThenBf eLi(p)if 

E^\f(xT t )-f(x Tt -)\<oo 

Ti<t 

39 for each t > 0. This certainly holds if / is bounded and E p (t, E) < oo. 
J J Bf(y, s, w)p{ds, dy) = J f(f(y) - f{x s ^))Q{dy; x s .)A(x s )ds 

o E [0,r] E 

+ J J(fiy) - f(x s -))Q(dy; x s .)ds s . (5) 

[0,f] E 

Suppose that / satisfies the boundary condition 



I 



/(*) = J f(y)Q(dy;x),x e T. (6) 

E 

Then the second integral in ^ is zero. The following result charac- 
terizes the extended generator A of (x t ). 

Theorem 2.1. The domain D(A) of the extended generator A of (x t ) 
consists of those functions f satisfying 
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(i) For each (n,z) e E the function t — > f(n,cp n (n,z)) is absolutely 
continuous for t e [0, t*(n, z)[. 

(ii) The boundary condition ^ is satisfied. 

(Hi) Bfe L l ° c (p). 

Then for f e D(A) 



A f{x) = Xf(x) + A(x) f[f(y) - f(x)]Q(dy; x). 

E 



(7) 



Proof. Suppose that / satisfies (i)-(iii). Then J Bf dq is a local martin- 
gale, and 

t i 

j Bfdq = Yj f(.xr t ) ~ f(x T - r ) - J f[f(y) ~ f(x s )]Q(dy; x s )A(x s )ds. 



E 



Now, 
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T t <t 



T,<t 



X /Oct,) - /(* Tj ) = £(/(*r,) " /^,- I )) + /(*,) - f(x Tn ) 

YSf^T-) ~ /(*T W )) + ft*) ~ f(XT n ) 



Ti<t 



where T„ is the last jump time before t. The first bracket is (f(x t ) - 
f(x )). Note that 

r, 
f{xr) - /(*r,_ i ) = I X" Vr ._ i /(v' r . t (f> VTj _ ] (f r,-_ i , s)ds a.s) . 

T-i 



So the second bracket is equal to J xf(x s )ds and 
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f Bfdq = f(x t ) - f(xo)- 

t t 

J f(f(y) - f(x s ))Q(dy, x s )A(x s )ds - Jxf(x s )ds. 

E o 

t 

So Af is given by (Q and CJ = J Bf dq. Conversely, suppose 

o 
/ e D(A). Then there exists a function h such that s — > h(x s ) is 

t 

Lebesgue integrable and M t = f(x t ) - /(xq) - J h(x s )ds is a local mar- 

o 
tingale. By the martingale representation theorem, M t = Mf for some 

g e L l ° c (p). Now the jumps of M t and Mf must agree, these only occur 
when t = Ti for some i and are then given by 

AM, = M t - M t . = f(x Ti ) - f(x T] ). 
AMf = Mf - Mf_ 



g(x„t,w)- I g(y,t,w)Q(dy,x^)I (Xt _ er) 



E 

at t = Ti. It follows that 

g(x, t, w)I (Xli/ r) = (f(x) - f{x t -))I(X t -tr) 
41 except possibly on a set G e E * p such that 

E y I IcP(dt, dx) = for all y e E. 

R + x£ 

Now suppose Xj. = z e T; then 

/(*) - f(z) = 8(x, t,o>)- J g(y, t, oj)Q(dy; z) 

E 

for all x except a set A e E such that Q(A,z) = 0. Since only the first 
terms on the left and right involve x it must be the case that 

f{x) = g(x, t, co) + f{t, OJ) 
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and 



f(z) = fg(y, t, co)Q(dy; z) + f(t, w) 

E 



for some predictable process /. Since g = f - f, 
f(z) = f f(y)Q(dy;z) 

E 

for z e T, i.e., / satisfies condition (ii). Hence 

g(x,t,o)) = f(x) - f(x t -). 

Hence we get 

\\(Bf-g)I it<(Tn) \\L l (p) = 0. 

So condition (iii) is satisfied. Fix co and consider (M f )„< r<<1 ( w ) start- 
ing at (v ,£,)-, then 



M t = f(v ,4> Vo (t,to))-f(v t 






h(x s )ds 



t 
Mf = j f(f(y) - f{x s ))Q{dy;x s )A(Xs)ds. 

o E 

Hence /(v o ,0 Vo (/,£,)) is absolutely continuous for t < T\(oS). Since 42 
(v ,£ ) is arbitrary and T\(w) > a.s. this shows that (i) is satisfied. 

A "Feynman-Kac" formula. 

This is used to calculate expected values of functionals such as 



E x 



I 



e as c(s, x s )ds + e fi(x t ) 



There is no extra generality in allowing a P.D. Process to be time- 
varying, because time can always be included as one component of % t . 
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However, it is sometimes convenient to consider the joint process (t, x t ) 

with generator A = \- A. Then for / e D(A) 

ot 

t t 

f(t,x t )-f(o,x )= I l—+A\f(s,x s )ds+ J Bfdq. 

O 

If I — + A\ f(s, x s ) = o and Bf e L\(p), then f(t, x t ) is a martin- 
\ds j 
gale, so it has constant expectation 

E Xo f(t,x t ) = f(o,x ). 
Then 



where 



f(o, x ) = E Xo <p{x t ) 
f(t, x) = cp(x) {(p prescribed). 



Proposition 2.5. Let t > o be fixed and a : [o, t] x E — > R.+, c : [o, t] x 
E —¥ R. anJ (f> : £ — > R be measurable functions. Suppose f : [o, t] X 
£ — > R satisfies: 



(i) /(j, . e D(A)) 
(ii) /(f, x) = <p(x), x e E 
(iii) Bf € L/Q7) 



(8) 



43 Then 



df(s,x) 
ds 



+ Af(s, x) — a(s, x)f(s, x) + c(s, x) = 
(s,x) e [o,t[ xE. 



(9) 



t I s 



f(o, x) = E (K 



I exp - I a{u,x u )du 



c(s, x s )ds 



( t 



+ exp 



- I a(u, x u )du 



<f>(x t ) 



(10) 
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Proof. Suppose / satisfies (|8}. Define 

I a(u, x u )du 



e s = exp 



Then 



d(e s f(s, x s J) = e s df(s, x s ) + f(s, x s )de s 

- e s I 1- Af \ds + e s Bf dq — a(s, x s )e s f ds 

\ds J 

= -e s c(s, x s )ds + e s Bf dq (by ®). 

Now by (iii), e s Bf e L\(p) since e s < 1. Thus the last term is a 
martingale and 



E x [e t f(t, x t ) - f(o, x)] = ~E X 



T 

I 



e s c(s, x s )ds 



This with (ii) gives (ITOl . 



Example 2.9. The Renewal Equation: 

Let {N t ) be a renewal process with inter arrival density /(.)■ Let 
m(t) = EN t . Since the process "restarts" at renewal times, 



E[Nt\Ti = s] 



|0, s>t 

\m(t - s) + 1, s < t. 



So 



-co-JW.-'i/w* 

which gives the renewal equation 

r 
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42 1. Stochastic Jump Processes 

This can be solved by Laplace transforms. Denning 

CO 

fip) = fe- pt f(t)dt 



etc., we get 

1 . 

m(p) = m(p) f(p) + - f(p). 
P 

So 

m(p) = t — . 

l-f(p) 

In particular, for the Poisson process f{t) = Ae~ M ", 

* A + p 

will give 

A 

m(p) = — 

P 1 

to get 



m(t) = At. 

Exercise 2.4. Compute M x {t) = E T N t , where the component in service 
at time has age x (and is replaced by a new component when it fails). 

(N t ) is a PD process if we take x t = (v t , % t ) where v t = N t and g t is 
the time since last renewal. Then 

X - 3 Atn - m 
~ d£ 7 ~ ~ 



J f(u)du 



45 and Q(.;v, £) = <5( V +i,o> so that 



A/(v, = ^/(v, + A(£)[f(v +1,0)- f(v, 0)]. 
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Use proposition [O] with a = c = and cf>(x) = v to get 

/(0,v,£) = E {vS v t . 
Clearly 

Define 

Then the equation for / (or /z) becomes 

-h(s,0 + — h(s,0 + A(m + Ks,0) - h(s,0] = (12) 
ds d% 

Define 

z(u) = h(u, u). 

Then 

d 

—z{u) = -A(u)[l + h(u,0) - z(u)] 

du 

Z(t) = 0. 
Thus z(u) satisfies 

z(u) = A(u) z(u) - A(u)[l + h(u, 0)] (13) 

where 

A(u 



F f f 

(«) = -— = —,F(u) = I f(s) ds. 



Equation ( fT3l is a linear ODE satisfied by z(u). The transition function 
corresponding to A(u) is 

00, v) = — — 
F(v) 

Hence dT3T > has the following solution at time 0: 46 
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t 

o 

Define 

m(s) = /z(7 - s, o). 

Then (TUT) coincides with the renewal equation (fTTT) . Having deter- 
mined /z(m, 6)o < u < t, h(s, £) for s + £, + o can be calculated from 
(fT2l . The result will be equivalent to that of Exercise 12.41 



Chapter 2 

Optimal Control of pd 
Processes 



General formulations of stochastic control problems have been studied 47 
using martingale theory, where the conditions for optimality, existence 
of optimality are derived (El Karoui fi"5l '). But this does not give ways 
of computing optimal control. Control of Markov jump processes has 
been studied using dynamic programming (Pliska dT\ ). In this Chap- 
ter, we will be dealing with control theory for PD processes, following 
Vermes E51 . 

Let 7 be a compact metric space. Control arises when the system 
functions X, A, Q contain a parameter y 6 Y i.e., for x = (v, £) 



xyf(x) = Y J b(v,t,y) 



df(v,0 



Q = Q(A,x,y). 

A feedback policy (or strategy) is a function u : R + x E — » Y. 
Let °i/ denote the set of all strategies, u is stationary if there is no t- 
dependence, i.e., u : E — > Y. Corresponding to policy u we get a PD 
process with characteristics X", A", Q u given by 



*7=y Kv,t,u(x))%(x) 



45 
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A u (x) = A(x, u{x)) 
Q u (A;x) = Q(A;x,u(x)). 

More conditions on u will be added when required. Then we get a 
PD process x t with probability measure P" determined by X u , A u , Q". 
48 Given a cost function, say, for example, 



J x (u) = E" 



I e~ as c(x s , u s 



,)ds + e- at (j>{x t ) 



where £" is the expectation w.r.t. P" starting at x and a > 0. The control 
problem is to choose u{.) to minimise J x (u). The "usual" approach to 
such problems is via "dynamic programming". Let V(s, x) be a function 
of (s, x). Introduce the Bellman-Hamilton- Jacobi equation 

dV(s.x) 

— ^ — - + min[A J V(j,i) + c(x,y)] - a V(s,x) = (B) 

OS y e Y 

where A y is the generator corresponding to X y , A(.,y), Q(.,y). 

If Y has one point, then this coincides with the equation for J x as 
before. 

Proposition 1. Suppose (B) has a "nice" solution (i.e., satisfies bound- 
ary condition etc.). Then 

V(o, x) = min J x (u) 

ue a // 

and the optimal strategy u"(s, x) satisfies 

A u " (s ' x) V(s, x) + c(x, u"(s, x)) = min (A y v + c). 

yeY 

Proof. Same calculations arise as before. Let x t correspond to an arbi- 
trary control policy u. Then 

d(e-° s V(s,x s )) = -ae- as V(s,x s )ds + e~ as (^- +A u v\ds 

+ e~ as BV dq > -e~ as c(x s , u(s, x s ))ds + e~ as Bu dq (1) 



47 
49 So 



V(o,x)<E x 



I 



t 

&s „/„ , , w„ . — at 



e c(x s , u s )ds + e <p(x t ) 



J x (u). 



Now suppose u = u , then "equality" holds in place of "inequality" in 
{Q>. So 

V(0,x) = J x (u ). 

So u° is optimal. n 

Objections: 

(1) There is no general theory under which (B) has solution. 

(2) u°(x) constructed as above may fail to be an admissible control: 
to make sense of it, we must be able to solve the ODE 

^-{(s) = K°({ s ) = b(v,t,u°(s,t)). 
as 

There is no guarantee that u° leads to a "solvable" ODE. 

So we must redefine "admissible controls" so that this is avoided. 

Remark 1. In control of diffusion processes, the equation is 

dx t = b(x t , u(x t ))dt + o~(x t )dW t . 

Here we "handle" nonsmooth u by using weak solutions. 

Remark 2. In deterministic control, one uses open-loop controls de- 
pending only on time. The equation here is of the form 

x = b(x t , u(t)). 

Then solution is well defined for the measurable w(.). 
Special cases: 50 
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(1) Control only appears in Q. Then the problem reduces to a se- 
quential decision problem where a "decision" is taken each time 
a jump occurs. (Rosberg, Varaiya and Walrand [22]). 

(2) X = 0. Here Markov jump process with piecewise constant paths 
are considered. Control appears in A and Q. 

Then 

( \ 



A u f = A(x, u{x)) 



/■ 

\E 



(f(z)-f(x))Q(dz;x,u(x)) 



is a bounded operator on B(E). Regard (B) as an ODE in Banach 
space B(E). Let V(s) := V(s, .), then 

dV 

— = g(V(s)) = mm(A y V + c). 
as yeY 

So g is a nonlinear function, but it is Lipschitz continuous in V 
[Pliska EDl- 

(3) Piecewise linear processes (Vermes [25 1). 

Here jj t is on dimensional and X = JL Control appears in A and <2. 
Consider a 'stationary' control problem, where the Bellman equa- 
tion takes the form 

min(A y V + c(x,y)) = 

yeY 

V(x) = O(x), xeE T . 
This corresponds to minimising 

/ T 



£, 



I C(X X , «. s 



)rfj + ®(Xr) 



V() 



where r is the first hitting time of some target set Ej. Then 

A>Y(x) = — V(v,f) + ^(x,y) J (V(z) - V(x))Q{dz, x,y). 

E 
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51 Suppose ve{ 1,2, ...,n) and 

Then Bellman equation takes the form 

^V(£) = g(VQ). 

This is an "ordinary" functional differential equation with non-stan- 
dard boundary condition. Vermes showed existence of an optimal feed- 
back strategy in special cases. 

'Generalised' Dynamic Programming Conditions: 

Let us consider next optimal control of the deterministic differential 
system: 

x, = f(x,,t,u t ), te[t G ,ti\. (2) 

Then the control problem is to 

n 
minimize I £{x t , t, u t )dt 

to 

over "admissible" control/trajectory pairs u t , x t i.e., pairs of functions 
for which 

(i) © is satisfied, 

(ii) x(t[) - xi, x(t ) = x with x , xi given, 

(iii) x t eA, u t eQ., where A, Q are compact and A = A x [t , t{\. 

This will be called the strong problem (S). 

We assume (a) an admissible pair (x t , u t ) exists, and also we make a 
temporary assumption 

(f(x,t,a)) 

(b) s i is convex. 52 

U{x,t,Q.)\ 
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This enables "relaxed controls" to be avoided. Define 
i7(S) = "value" of S 



,/' 



i.e., inf | £dt. 

(x,,u,)ad , 

Theorem 1. There exists an optimal admissible pair (x t , u t ) for the 
strong problem. 

This is a "standard" result in optimal control theory (Vinter and 
Lewis [26]). It depends critically on the convexity assumption (b). 

A Sufficient Condition for Optimality: (Standard Dynamic Program- 
ming). Suppose (x t , u t ) is admissible and O is in C l (A) such that 

<3>,(t, x) + max(0 A: (x, f)f(x, t, u) - £{x, t, u)) = 

0(x/, t{) eAxQ. 
and $> t (t, x t ) + 0(x ? , t)f(x t , t, u t ) - t(x t , t, u t ) = a.a.t. 

then (x t , u t ) is optimal and r](S) = -0(x , t () ). The main result of Vinter 
and Lewis is as follows. 

Theorem 2. The strong problem has a solution (i.e., there exists an 
optimal pair (x t , u t )). There exists a sequence {<D'} in C l (A) such that 

Of + max(O x / - €) < o, (x, i)eA 

ueQ. 

O ! '(xi,?i) = 
53 and (x t , u t ) is optimal if and only if 

YimH\t) = in Li\t ,t\\ 

t— >oo 

where H'(t) = <3>' t (x t , t) + ®> l x {x t , t)f(x t , t, u t ) - £{x t , t, u t ). 

The Weak Problem 

For (x t , u t ) admissible, define p xu eC*(A x Q), the dual of C(A x Q), 

by 

n 

< g,Hx,u >= I g(x t ,t,u t )dt 

to 

for arbitrary geC{A x Q). p xu satisfies 
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(i) p x , u eP + (i.e., if g > o then < g,/u x , u >> o). 
(ii) Take (peC l (A) and g(x, t, u) = <p(x, t) + (p x (x, t)f(x, t, u) 
then 

< <Pt + </>xf,Mx,u >= 4>(XI, t{) - (f)(x , t ). 

Define 

yu = {/j.eC*(A x D.) : (i) and (ii) are satisfied}. 

Proposition 2. fx is weak* compact and convex. 

Note 1. The cost function for (x t , u t ) is < t,fi x ,u >■ 

Weak Problem (W): Minimise < (, u > over jue^. So 54 

J](W) < 7/(5). 

Theorem 3. n(S) = rj(W). There exists an optimal x, ufor S, so fi XM is 
optimal for W. 

Now we incorporate the constraints on [i into the cost function in the 
following way. Define extended real valued functions p, q on C*(A x £1) 
as follows: 

\<€,p> if \fx\<ti-t ,fieP + 
\+oo otherwise. 

Let ^#2 = {peC* : Condition (ii) is satisfied}. Then 

_ (o ifpeM 2 
l-oo otherwise. 

Proposition 3. p is I. s.c. and convex, 

q is u.s.c. and concave, 
and 

rj(W)= M{pQi)-qQi)}. 
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The Fenchel dual problem is as follows: 

max {q\0-p\0) (P) 

£eC(AxCl) 

where p* , q* are "dual" functionals defined by 

p*(£) = sup <£,p> -pip). 

q*i{)= inf <f,p>-qi M ), 
/xeC* 

Proposition 4. 

p*i%) = max (f it, x, u) - lit, x, u)) + x 0/ - t () ). 

(x,t,u)eAxQ. 

55 where a + := max(<a;, 0). 

Sketch of proof: 

p*(f)= sup [<£,p> -<{,p>] 

\fl\<tl-to 

= sup [<£-£/*>]. 

\n\<H-t 
lieP + 

If < %,p > - < I, p. > is negative, then the optimum is zero. If 
< %,p. > - < t,p >> o, then put Dirac measure xiti - t () ) on maximum 
point to get the result. 

Proposition 5. 

W = {%eC :% = $, + (pxffor some (peC\A)}. 

Then 

(-00 if% $ W 

q i0 = mmtf(xuh) ~ <PXx , to)) ifteW 

where i; = lim i; and g = <f>\ + <f>' x f. 
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Proof. For %eW, by definition of q and q* , we get q*(%) = (f>(xi,t\) - 
4>(xo, t ). 

A similar argument gives the result for ueW. For g <£ W, there 
exists a separating hyperplane, i.e., jleC* such that < g,ju >+ 0,^eW 
and < £,yu >= 0. If \xe^2, then /j + cJleM2. So 

?*(£) = inf <^,yU>=-oo. 

/1£M 2 
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77(D) = max 



Yim(<p\xuh) - <p\x ,t o )) 



- max (£(/, x, m) - ^(f, x, u) + (ti, t )) 

(x,t)eAx.Q. 

= s\xp(<f>(xuh) - <p(x , t ) - max(0 r + (p x f - {) + (t t - t )). 

It is no restriction to assume <f>(x\,ti) = 0. Then Vinter and Lewis show 
by an ingenious argument that 

77(D) = SUp(-0(x„, t )) 

where the supremum over cpeC 1 such that (p(x\, h) = and (<p t + tp x f - 
€) < V (x, t, u). 

Theorem 4. 

77(D) = r0T) = 77(5). 

Proof. This follows from a "standard" result in duality theory: q* is 
finite at some point in its domain where p* is continuous. □ 

Proof of the main results now follow easily. The strong problem has 
a solution since 77(5 ) = 77(D). 

i 1 {S) = \im{-(P\x ,t )) 

for some sequence of 0"s satisfying the "Bellman inequality". The char- 
acterization of optimal pairs (x t , u t ) follows. 
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Remark 3. If the set 57 

jf(x,t,0)\ 

\€(x,t,Q.)) 

is not convex, them the results are still valid but relaxed controls must 
be used. 

A relaxed control /u t is a C*(£2)-valued function on [t , t\] such that 
fi t is a probability measure for every t and t — > f g(t, u)/j. t (du) is measur- 
able for every continuous function g. 

Interpretation: x t , fi t is an admissible pair, whenever 

f(x t , t, u)/j. t (du); 



dx t r 

dt ~ J ' 



n 
h 

the cost is I I f(x t , t, u)fi t (du)dt. 

to fi 

Optimal Control PD Processes (Vermes |25|) 

In this section, we adopt a slightly modified definition of the PD 
process (x t ). It will take values in E, a closed subset of R , and we 
suppose that 

E = E UEqUEj (disjoint) 

where Ej is a closed set, E is an open set and 

E d = (E -E )-E T . 

Let E' Q , E' E' T be compactification of E , Eg, Ej respectively and 
£" be the disjoint union of E' , E' d , and E' T . Then a controlled PD process 
is determined by functions 

f:E' xY^> R d ; 
A: E' l} xY -> R+, 
and Q : (E' a UE') x Y -* m x (E ) 

58 where m/(£ ) is the set of probability measures on E and Y is a compact 
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(control) space. Function / gives the deterministic motion by 

x t = f(x,,y,). 

We assume / satisfies a Lipschitz condition in x. 
Admissible Controls: Feed back controls u(t) = u(x t ) are not the 
"right" class of controls because the equation x = f(x, u(x)) only has 
a unique solution under strict conditions on «(.). Let 

a t = last jump time before t. 
n(i) = x a {t) 
z(t) = t - ait). 

Then n(t), z(t) determine x t ; in fact, for fixed yeY, 

x t = ^-n(t),z(t) 
z 

where X n<z = n + I f(X n<s ,y)ds. 



Then admissible controls are y-valued measurable functions 
u(n(t),z{t)). By Caratheodory's theorem, the equation 



z 

X n , z = n + I f(X ntS , u(n, 



s))ds 



has a unique solution, and PD process is well defined for such u. We 59 
will consider the three component process (x t ,Zt, n t ) for notational con- 
venience. 

Relaxed controls are functions [i : E x R + — » m\{Y) such that (n,z) — > 
J (f>(n, z, y)fi(dy; n, z) is a measurable function for (n, z) for all continuous 
(p. Corresponding to /i, define 

f{x,n,z)= I f(x,y)fi(dy,n,z) 

A^(x,n,z)= \ A(x,y)/u(dy;n,z) 
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! 



&{A,x,n,z) = I Q(A,x,y)p(dy,n,z). 

Then we construct a PD process (x t ,n t ,Zt) corresponding to jP", ,V, 
<2 P in the usual way. 

The strong problem is to minimise J% o (ju) over admissible relaxed 
controls p., where x = (x, x, o), and 



( T 



J*M = K 



I I i {x t ,y)p{dy,n t ,zt)dt 

■.o Y 

+ /j I to(xt-,y)Kdy,n t -,z t -) + £rOr) 

{;:*,_ e£ fl I J 

Here t is the first hitting time of set Ej. 

Main Results: 

Theorem 5. There exists an optimal (relaxed) control. 

Theorem 6. The value function ifr(x) = sup <f>(x, x, o) where the supre- 
mum is over all functions 0eC l (E) such that 



<f> z {x, n, z) + min V x (p(x, n, z)f(x, y) + A(x, y) 

yeY \ 

if <f>({,t,o)Q(df,x,y) - <Kx,n,z)\ + € {x,y)\ ix.,,,z)eE„ (3; 

>0 

<p(x,n,z) <mm| f $(£,£, o)Q(c%,y,x) + t (x,y)\(x,n,z)eE a (4) 

(f>(x, n,z) < (t(.x), xeEj (5) 

60 and E is the space of triplets (x, n, z). 

Theorem 7. There exists a sequence <f> satisfying @, (0), @ above 



such that p° is optimal if and only if 

<f> k (x, n,z) + J j V,/(x, n, z)f(x, y) + A(x, v)) 
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I 



<p k (t,Z,0)xQ(dt,x,y)-<p k (x,n,z) 



+ Ux, y)fi°(dy, n,z)^o in U{Q° ). (6) 



u 



4> k ({,f,o)Q(d{,x,y) + ( (x,y) 



fi°(dy, n, z) 



-<f>\x,n,z)^OinL l (Q° g ) 
cp k (x,n,z) - t T (x) -» in LiCQj). 

The measures Q°, Ql and Q' T are defined as follows. 
Denote x t = (x t , n t , Zt)- For AeE , 



02(A) = Eg n(x,)dt 



T 



(7) 
(8) 



which is a measure on E and is called potential measure ofx t . 
Q°(A) = E^^Aix,.) 
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where A 6 Eg. 



for A e E T . 



Q°(A) = P t i°[x T eA] 



Comparing with deterministic case, the necessary and sufficient con- 
dition there was that (x t ,fit) is optimal if and only if 

</>' t (x t , t) + I [cf> x (x t , t)f(x t , t, u) - t(x t , t, u)]fi t (du) -> o in L/(f , h). 

The "probability measure" corresponding to p. t is Dirac measure on 
x(.) and Qg(A) is the time spent by x(.) in A. Thus the conditions stated 
are a direct generalization of the deterministic ones. 



Remark 4. Note that if we define 
Q°o(A) = E- x 



f 

Jo 



\f/ A {x s )ds 
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then for any positive measurable function g, 

E x [ g(x s )ds = j g (t)Q'M); 

for, if g{x) = 2] CiiffA,(X) 



62 then 



( ( 

Ex I g{x s )ds = 2_j C[E X I iff Al (x s )ds 

o ' 

i 

E„ 

The general case follows by monotone convergence. 

Remark 5. The Q° are "potentials of additive functionals" ( t is an ad- 
ditive functional if I > o anqj 

C t+S = €, + € s o9 t 
t, p* , I(t> T ) are some example of additive functionals. 

The potential of an additive functional is an operator 



T 

fg(Xs) 



U e g(x) = E x g{x s )d€ s . 



Here Q° Q , Q°, Q° T correspond precisely to this with C, = t,p*,I( t> T) 
respectively. 

The Weak Problem: 



l 8, is the shift operator on the space of right continuous functions: (8,w) s = u>, +s 
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The deterministic weak problem involved the fact that 



(f>(xi,t\)-<f>(x t 



■o, to) = I (<f>t 



+ (p x f)ds 



'„ 



for any <p e C . The stochastic equivalent of this is Dynkin's formula. 
To get this in the appropriate form, define operators A y , B y as follows. 

A y (p(x, n, z) = (p z (x,n,z) + v x <p{x,n,z)f{x,y) 

+ A(x, y) f(cf>(t, £ o) - <f>(x, n, z))Q(d{, x, y) 

E 

and B>^,n, z > = fM,?,oMK*,y>-«*,n^ 



for (x, n, z) e Eg. Then the Dynkin formula on the interval (o, t) is 63 

E?Mx T , tir, z T ) - <p(x) 



Now 



- ^ 



// 

o Y 



A y <p(x t , n t , z t )fi(dy; n t , z t )dt 



+ I I B y (/>(x t ,n t ,ztMdy,z t ,n t )dp* 

o Y 

= I I A y <p(x,n,z)n(dy;n,z)Qf ) (dx,dn,dz) 

£„ y 

+ JJB y (K x,n,zMdy, X ,z)QZ(dx,dn,dz) 

E d y 

E^<f>(x T ,nT,ZT) = I 0(x,n,z)Q^(dx,dn,dz). 

h. i 
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So we can express the Dynkin formula as follows: 
(p{x Q )= I L(p{x,y)M^{dx,dy) 

ExY 

where L<p{x,n,z,y) = i//g T (f>(x,n,z) + if/g g A y (f>(x,n,z) + \p^B Y {x,n,z). 



M tl (S l xS 2 ) = Q' 1 T (Sif]E T )+ f f n(dy;n,z)Q%(dx,dn,dz) 

I I ji{dy;n,z)(t d {dx,dn,dz). 



SiDE,, s 2 

+ 

SiDE d s 2 
64 The cost for the relaxed control p is 



J-^) = f l(x, y)M^(dx, dy). 



ExY 

The following supplementary assumption is required. 

inf J io {p) = inf J io {p) 

for some c> o and u is the set of relaxed controls, 
u c = {p e u : p e [r + p*] < c\ 
with this assumption the weak problem is to minimize J IdM over 

ExY 

measures M e m\ +c (E x Y) (where m a is the set positive measures of 
total mass less than or equal to a) such that 

1. M = M + M d + M T 

where M T e m x (E T ). 

Mq e m(E d x Y) 
M € m(E a x Y). 
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2. (f>(x ) = fL(pdM,(t>e C l (E). 

65 From this point on, the development follows the Vinter-Lewis argu- 

ments closely. We reformulate the weak problem as a convex optimiza- 
tion problem by incorporating the constrains in the cost function and 
obtain the characterization of optimality by studying the dual problem. 
The reader is referred to Vermes [25 ] for the details. 

Remark 6. The optimality condition involves the measures Q° , Q" Q" T 
corresponding to //". These can be computed from the following system 
of equations. 

A^"h{x) + (Ar n e„ = 0, x e E 
B»°h(x) + ift Tneg =0,xeE d 
h{x) + ip vr]£T {x),xeE T . 



Then 

Q°{T) = h(x ). 

Example 1. If T c E , then Dynkin's formula says 

T 

h(x ) = E\ q I i//r(x.,)ds 
o 
= Q° (D- 

The results outlined above are the first general results on optimal 
control of PD processes. Obviously much work remains to be done; 
natural next steps would be to determine necessary conditions for op- 
timality of Pontrjagin type; to develop computational methods; and to 
study optimal stopping and "impulse control" for PD processes. For 
some related work, see van der Duyn Schouton [29], Yushkevich [30] 
and Rishel [311. 



Part II 



Filtering Theory 
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0. Introduction 65 

Introduction 

Suppose {x t \ is a signal process which represents the state of a system, 66 
but cannot be observed directly. We observe a related process {y,}. Our 
aim is to get an expression for the "best estimate" of x t , given the history 
of {y t \ upto time t. 

In Section ^ we give quick derivations of the "Kalman filter" for 
the linear systems, and nonlinear filtering equations, that of Fujisaki, 
Kallianpur and Kunita and Zakai's equation for unnormalized condi- 
tional density (Kallianpur [19|, Davis and Marcus [8|). In section |2j 
we will study pathwise solutions of differential equations. In section 
|3j we will study the "Robust" theory of filtering as developed by Clark 
1 5 1, Davis [10 1 and Pardoux [20]. Here the above filtering equations are 
reduced to quasi-deterministic form and solved separately for each ob- 
servation sample path. Also, we will look here into some more general 
cases of filtering developed by Kunita IIT71 . where the existence of con- 
ditional density functions is proved using methods related the theory of 
"Stochastic flows". 

1 Linear and Nonlinear Filtering Equations 
Kalman Filter (Davis flH) 

Suppose that the "signal process" x, satisfies the liner stochastic differ- 67 
ential equation 

dx t = Ax t dt + cdV, (1) 

where V t is some Wiener process. "Observation" y t is given by 

dy, = Hx t dt + dW t (2) 

where W t is a Wiener process independent of V, . Assume x () ~ N(o, P ). 
To get a physical model, suppose we write (J2ji as 

dy, „ dW t 

— = Hx t + — — 
dt dt 

dW t dy t 

then corresponds to white noise and — is the " physical" obser- 

dt dt 

vation. 
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The filtering problem is to calculate 'best estimate' of x t given 
(y s , s < t). There are two formulations for Kalman filter. 

(a) Strict Sense: If (V t ,W t ) are Brownian motions then (x t ,y t ) is a 
Guassian process. Then x t = E[x t \y s , s < t] is the "best estimate" 
in the sense of minimizing E(x t - z) 2 over all y r measurable, square 
integrable random variables z, where 

y, = cr{Y s , s < t). 

Because of normality, x t is a liner function of (y s , s < t). 

68 (b) Wide Sense Formulation: Do not suppose V t , W t are normally dis- 
tributed. Just suppose that the i coordinates V' t , W' t are uncorrelated 

and EV' t V' s = tAs; EW' t W' s = tAs, i.e., V, W are orthogonal incre- 
ment processes. Now look for the best linear estimate of x t given 
(y s , s < t). This will coincide with E(x t \y t ) in strict sense case. 

Calculating x t is a Hilbert space projection problem. The random 
variables we consider belong to L^O, F, P) which is a Hilbert space 
with inner product (X, Y) = EXY, where o denotes the elements are 
of zero mean. For any process, say y t define H y = L(y t , t > o), the 
linear span of y t ; this is a linear subspace. Then if z denotes the 
projection of z onto H y , then 

||z- £|| = min ||z - U\\. 

UeH> 

Let x t be projection of x t onto H y = L(y s , s < t). Then the "Innova- 
tions process" v t is defined by 

dv t = dy t - Hx t dt. (3) 

The Innovations process v t has the following properties: 

(i) v t is an orthogonal increments process. 

(ii) H] = HJ. 



(iii) HJ = lfg(s)dv s , g € L 2 [o,t]\. 



I . Linear and Nonlinear Filtering Equations.. . . 67 

69 Then x t satisfies the linear equation 

dx, = Ax t dt + P{t)H'dv, 
x o = (4) 

where the error covariance P(t) = E(x t - x t )(x t - x t )' , (' denotes the 
transpose). 

P(t) satisfies the "Riccati equation" 

—P(t) = AP(t) + P(t)A' + CC - P'(t)HH'P(t) 
dt 

P(o) = P = Cov(x ). 

The above equation © is the Kalmcin Filter. 

Derivation of Kalman Filter equation: From properties (ii), (iii) we 

know 

t 

x t = I g(t, s)dv s 

o 

for some g such that 

t 

I g 2 (t,s)ds < oo. 

o 

Now using projection, x t - x t J. v s , s < t. So 

Ex t v' s = Ex t v' s 

( ' 

= E\ | £U, w)dv„ v\. 



I g(t,u)dv u 
I g(t, u)du. 



Hence 

g(t, s) = —Ex t v' s . 
ds 
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Write innovations process as 70 

dv, = Hx t dt + dW t where x t = x t - x t . 

s 
Ex t v' s = I E(x t x' u ))H'du. 



Now 



x t = (p{t, u)x u + I <p(t, r)CdV r 
u 
where <p is the transition matrix of A. So 

■s 

Ex,v' s = I cp(t, u)(x u x' u )H'du 

<> 

s 
= I (p{t,u)P{u)H'c 



So 



du 

g(t,s) = <f>(t,s)P(s)H'. 

t 
x, = I <p(t, s)P(s)H'dv s . 

o 

But this is the unique solutions of ©. 
Important Points: 

(1) It is a recursive estimator. 

(2) In the strict sense version x t is a sufficient statistic for the condi- 
tional distribution of x t given (y s , s < t), since this distribution is 
N(x t , P(t)) and P(t) is nonrandom. 

Exercise 1.1 (Constant Signal). Let x, = 6 with E(8) = 0, Var (8) = <x 2 
and 

dy t = 6dt + dW t 
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71 with 6 independent of W t . Show directly by projection that 

1 



ft = — T-y t - 
t + \ 

Now show that the Kalman filter gives the same result. 

Nonlinear Filtering 

Suppose "signal" x t is a Markov process and "observation" y t is 
given by 

dy t = h(x t )dt + dW t , 

generally h is a bounded measurable function (extra smoothness condi- 
tion will be added later). Assume that for each t, x t and (W u - W v ), u, v > 
t are independent, which allows for the "feedback" case. Our objective 
is to calculate in recursive from the "estimates" of x t . to do this, it is 
necessary to compute the condition of x t given 

y y = cr{y s , s < t). 

The Innovations Approach to Nonlinear Filtering 

This approach was originally suggested by Kailath for the linear 
case and by Kailath and Frost for nonlinear filtering. The definitive 
formulation of the filtering problem from the innovations standpoint was 
given by Fujiskki, Kallianpur and Kunita (W\ . 

Innovations Processes: Consider process y t satisfying 72 

dy t = z t dt + dW t , t e [o, T] (5) 

where W t is Brownian motion and assume 

T 

E J \ 2 s ds < oo (6) 

o 

and the "feedback" condition is satisfied. Let 

it = E[z,\y t ]. 
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More precisely % is the "predictable projection" of z t onto y t . The 
innovations process is then 

dv t = dy t - iydt. (7) 

Note (i): v t is a Brownian motion w.r.t. y t , i.e., v t is a y t martingale and 
< v > t . If F v t = cr{v s , s < t), the question is whether F y t = F y t . It has 
been shown that in general, this is not true. But if (i) holds and (z t ), (W t ) 
are independent, then Allinger-Mitter proved that F\ = F y t . 

Note (ii): All y r martingales are stochastic integrals w.r.t. (v t ), i.e., if 
M t is a y r martingale, then there is a g such that 



I 



g s ds < oa a.s. 



and 

t 



'"I s ' 



M, = s x dv s ■ 



o 

This is true even if F v t + F y , but note that (g s ) is adapted to F y t , not 
necessarily to F y . 

73 A General Filtering Formula: Take an F r martingale n t , process (a t ) 
satisfying 

T 

\ 2 ds < CX3 



Jm 



and F () measurable random variable % with Ei& < oo. 
Now define an F t semi-martingale % t by 



& = £, + I a s ds + n t . (8) 



a 
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Since < W, W > t = t, we have 



T 

n, W >,= I fi s c 



for some (3 t and for any martingale n t . Let 

It = m\yt\. 

Then | f satisfies the following stochastic differential equation 
t t 

|, = i (J + J a s ds + I [£z s + 1,£, +ft s ]dv s . (9) 

o o 

Proof. Define 

t 

Ht = it- h> = I a s t 



t 
\ds 



Then /j. t is a y t - martingale. So there is some integrable function rj 

such that 

t 

fit = I n s dv s . (10) 



Now we will identity the form of n t , using ideas of Wong [ 28 1 . Using 74 
^ and (JSJl and I to formula, 

t t 

&yt = € y + I €s(z s ds + dW s ) + I y s {a s ds + dn s ) 

O 

Now calculate $ t y t using Q and (ITOl . 

t t t 

hyt = | y + I £ s (z s ds + dv s ) + I y s (a s ds + rj s dv s ) + I n s ds. 
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Now for t > i 


> 

e [&yt - ity t \y s ] = 0. 


So 


E 


1 ((&£«) - lulu +Pu- T] u )du\y s 




- s 



0. 



Let 

V(U) = guZu ~ lulu + Pu ~ rju- 

Then V(u) is predictable process and 



f V(u)du\F s 



0. 



This is, 



I v< " 



)rf«rfP = OVi, i>i,Aef s . 



Ax[s,r] 



The class of sets Ax [s, t] generates P, the predictable cr-field. Hence 
V(u, to) = a.e. dt * dP. Hence the result. 
75 Formula © is not a recursive equation for | f . Still we can use it to 

obtain more explicit results for filtering of Markov processes. Let (x t ) 
be a Markov process and A, D(A) be generator, i.e., for / e D(A) then 

r 
C{ = f(x t ) - f(x s ) - j Af(x u )du 
s 

is a martingale. Suppose 

r 
< c f , W >,= j Zf(x s )ds 



for some function Zf. Introduce the notation 

Y\(f) = E[f(x t )\y t ] 
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Now apply © with % t = f(x t );Af(x s ) = a s , C t = n, and z t = h(x t ) 
to get Fujisaki-Kallianpur-Kunita filtering formula 

t t 

n f (/) = IW) + J U s (Af)ds + J [U s (Df) - U s (h)U s (f)] dv s (11) 

o o 

where 

Df(x) = Zf(x) + h(x)f(x). 

If we interpret n f as the conditional distribution of x t given y t , so 
that 



n,(/) = J 



f(x)U t (dx) = E[f(x t )\y t ], 



then (ITTl is a measure-valued stochastic differential equation, and gives 
an infinite-dimensional recursive equation for filtering. 

Exercise 1.2. Derive the Kalman filter from the Fujisaki-Kallianpur- 76 
Kunita equation. 

The Unnormalized (Zakai) Equations: 

Introduce a new probability measure P a on (D, F) with £ e [o, T] by 



dP 



( T T \ 

fh(x s )dW s -^fh\x s )ds 



Since h is bounded, P is probability measure and (v f ) is a f - 
Brownian motion. Also 

< c{,y > t =< cf,w > t 



t 

I Zf(x s )ds. 



Note that, in general, C t is a semi-martingale under P but < ., . > is 
invariant under absolutely continuous change of measure. Also if Z = o, 
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then x t has the same distribution under either measure. Let 

( t t ^ 

is 



dP 



I h(x s )dy s - - I h 2 (x s )ds 



Let E denote the expectation under P . Then it can be calculated 
that under measure P , 

IW) = E[f(x t )\y t ] 
£ [A t |y t ] 

' o-,(l) ' 

77 Then cr,(/) is an unnormalized conditional distribution since cr t (l) 

does not depend on /. To obtain ajn equation satisfied by <j t , we need a 
semi-martingale representation for cr t (l). First we have 



dA t = h(x t )A,dy t 



(12) 



i.e., 



I 



A t = 1 + I h(x s )A s dy s 

also A f is a (F,, P„) martingale. Then as before 

A, = E [A t \y t ] 
is a v f -martingale, so there exists some y r adapted integrand r\ t such that 



A r = 1 + r] s dy. 






(13) 



To identify r\ u we use the same technique as in deriving the FKK 
equation. Calculate using (fl2l and I to's rule, 



A,y f 



t t t 

I A,dy s + I y s A s h(x s )dy s + I A s h(x s )ds. 
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Calculating using ( fT3l and Ito's rule, 



Now 



so we get 



IT T 

A t y t = I A s dy s + I y s Tj s dy s + I n s ds. 



E [A t y t - A t y t \y s ] = o for t > s, 



r] t = A(t)h(x t ) := E [A t h(x t )\ytl 
So (fT3l becomes ag 



A t = I + f A s U s (h)dy s 
This has a unique solution 



t t 



A t = exp 



fu s (h)dy s -jju 2 s (h)ds 



= cr,(l) 
Theorem 1.1. cr,(/) satisfies the "Zakai equation" 

do-tif) = o- t (Af)dt + o- t (Df)dy t 
o- {f) = U (f) = E[f(x t )]. 

Proof. Direct calculation using (fTTT i. (TrU i and the fact that 

cr f (/) = ATW). 
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(14) 



(15) 



Corollary 1.1. There is a one-to-one relation between Zakai equation 
and FKK equation, in that whenever o~ t satisfies Zakai equation, o~f(f)/ 
o~ t (i) satisfies ftllt . and whenever Ylt(f) satisfies Alii . 



t t 



IW)exp 
satisfies Zakai equation. 



f U s (h)dy s - j f U s (h)di 



76 



The Zakai Equation with Stratonovich Integrals: 79 

Recall 

t t 

I u s o dv s := I u s dv s H — < u, v > t 

o o 

where o denotes a Stratonovich Stochastic integral and u and v are con- 
tinuous semi-martingales. We have to calculate < cr.(Df),y > t . From 
Zakai equation 

d<r t (Df) = a t (ADf)dt + <r t (D 2 f)dy t . 

So 

d < (r.(Df),y > t = a- t {D 2 f)dt. 

So the Stratonovich version of the Zakai equation is 

d(r t {f) = cr t (Af)dt + <r t (Df)ody t - l -a,(D 2 f)dt 
= o- t (Lf)dt + cr,{Df)ody t 

where 

Lf{x) = Af{x) - l -D 2 f{x). 

Application to Diffusion Process: 

Consider a process x t e R d satisfying 

df( Xt ) = X f(x t )dt + Xjf(x t )odB J t (16) 

for arbitrary smooth /, where X , . . . ,X r are vector fields on W 1 we sup- 
pose that < BJ, W > t = ah for some constants a 1 ,. . . ,a r 
80 Note that A is the generator of x t under measure P (not P ). This is 

given by 

Af(x)=X f(x) + ^X 2 f(x). 

i 

Proof. Rewrite (fT6l in Ito form. Replace / by X^f in dl6l . 
dX k f(x t ) = X X k f(x t )dt + XiX k f(x t )odB j t . 
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Then 

d < X k f, B k >,= X 2 k f{x t )dt. 

Then Ito version of ( fTol l is 

df{x t ) = ix o + 1 ~Yj X j) f^ dt + x jf( x t)dBJ. 
So 



anc|l 


A=X + -Y J X) 

t 




C{ = jxjf(x s )dB j s . 

o 


Propof 


iition 1.1. For Z given by 




< C f , W >,= ( Zf(x s )ds. 



with Zf = cr J Xj,Z is a vector field. 
Proof. 



d<C f ,W> = d< ( XjfdB', W > 



= Xjfd < B j , W > 
= a j Xjf{x,)dt. 

So 81 

D = Z + h 

= a'X, + h. 



2 We sometimes use the convention of implied summation over repeated indices. 
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Proposition 1.2. There exist vector fields Y Y\,... Y r such that 



A - -D 2 = - y Y 2 + Y - -Dh. 

2 2 ^ J 2 

i 



Proof. 



D 2 f = (a% + h)(a j Xjf + hf) 

= a'a-iXiXjf + a%{hf) + a j hXjf + h 2 f 
= a'a-iXiXjf + hZf + Dhf. 

a 

Let a = (ff'o: 2 , . . . a r )' and suppose / - aa' is nonnegative definite. 
Write (7 - aa') = AA' and let X = (X u . . .,X r )'. 

X'AA'X = Y J X 2 - rfaJXjXj. 

So define Y = A'X. Then Y'-s are vector fields and 

A - -D 2 = - y Y 2 - hZ + X - -Dh 

2 2 ^ ' 2 



? 1 , 

Yf + Y n - -Dh 



where Y a = X - hZ. It remains to check that / - aa' > o. Take £, e 
withal = 1. Then 



f (/ - aa'){ = 1 - {a'tf 
> l|c 
>0. 



> 1M 2 



82 Since \a\ 2 < 1, for 

< V a l B\ W > t = ^ a i < B '> W > t 



\a\ 2 t 
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So a 2 t = EUW„ where U = £« a l B\ 

\a\ 4 t 2 < EU 2 t = \a\ 2 t 2 . 
So we have Zakai equation in Ito's form 

d(r,(J) = o- t (Af)dt + cr t (Df)dy t 
With vector fields Xq,X\, . . . X r and in Stratonovich form 

do- t (f) = cr t {Lf) + cr t {Df)ody, 

With vector Fields Yq,Y\,... Y r plus a "O th -order" term. Now we 
investigate what sort of process is x t under the measure P . 

Proposition 1.3. Under P , x t satisfies the equation 

df(x t ) = Y f(x t )dt + Zf(x t )ody t + Yjf(x t )odb J t (17) 

where b ,. . .b r are independent standard Brownian motions indepen- 
dent ofy t 

Proof. Recall the Girsanov transformation. Under P,B l ,...B r aie inde- 
pendent and < BJ, W > t = ah. Now 

dP„ ( 1 

-^■ = exp\M t -j<M,M> t 

where M is a f-martingale. Under P , Bj- < B\ M > t is a martingale 83 
and hence a Brownian motion. Here 

t 
M t = - fh(x s )dW s ; 

O 

d < B j ,M >= -a j h(x t )dt. 
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So under P , 

dVJ = dB J t + a j h(x t )dt 

are independent Brownian motions, but V 7 is not independent of y t , in 
fact < V j ,y > t = ah. 

Now define bj = V/ - ajy t , then < b*,y >= 0, and this implies b J ' ,y 
are independent. But the & are now not independent. In fact, 

I -ai(x k t for k + j 



<b J ,b k > t -- 



1 - (aJ) 2 t for k = j 



So 



<b j ,b k >t = [<b j ,b k > t ] 
= (I - aa')t. 

Let (7 - aa') = AA' as before and define b t = A~ l b t . 
Then b t = A'b y and < b >,= It. So 

df(x t ) = X f(x t )dt + Xjfix^odB 1 

= X f(x t )dt + Xjf(x t )o(-a j h(x t )dt + a^dy, + dbj) 

= (X f(x t ) - hZf(x t ))dt + Zf(x t )ody t + Yjf(x t )odbj 

= Y f(x t )dt + Zf{x t )ody t + Yjf(x t )o d b\ 

where Y f = X f - hZf and Y = A'X. 

The so called Kallianpur- Striebel Formula gives the solution o~ t of 
the Zakai equation as a function space integral in the following form, 
where x t is functional of (y, b l , . . . b r ). 



o- t (f) = E 



f{x t ) exp 



/ t t 

I h{x s )dy s - - I h 2 {x s )ds 



yt 



i.e., as a function of y, we have 



f{x,) exp 



I h(x s )dy s - - I h 2 (x s 



<r,(f)(y)= J 

C[o,t] 

where n w {db) is the Wiener measure on C[o, T]. 



)ds 



H w (db l )...n w (db r ) 
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2 Pathwise Solutions of Differential Equations 

Consider the Doss- Sussman construction for the equation 

x = b(x) + g(x)w (1) 

x(o) = X 

where w e C'(R+). Let <p{t, x) be the "flow" of g, i.e., 

d 

-<p(t,x) = g(<f>(t,x)) 

(p{o, x) = X. 
If b = o, then it is immediate that the solution of Q is 85 

x t = <p(w{t),x). 
\ib + o, then the solution of Q is of the form 

xt = <f>(w(t),J](t)) (2) 

where 77(f) satisfies some ODE. With x(t) defined by (J2J), 

X(t) = g(x(t))w(t) + <Px{w{t), 7](t))j](t) 

and we require that 

(f> x (w(t),T](t))fi(t) = b(4>(w(t),n(t))). 
So x(f) satisfies Q if n(t) satisfies 

77 = (<f> X (w(t),T](tW l b(c/>(w(t),I](f))) 
77(0) = X 

Coordinate-free form: Let 

df 

X () f(x) = b(x)-f- 

dx 

df 

Xif(x) = g(x)-f- 

dx 
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£(*) = <p(w(t), X). 
Define 

(&x )f(x) = x (M; l )(% t (x))- 

Then the equation for rj(t) can be expressed as 



d 
/( 

Vo = x, 



j t m) = (c lx o)f(it) 



86 for 



(^X G )f( m ) = b{z t {x))j- x f{t;\x))\s, ix) 

= m{x)) ah f{x) Tx { ^ {x))k,{x) 

= b^ t {x))^-f{x){Ux)r l . (*) 

ax 

Since £ _1 (£(x)) = x and so 

d , <i 

— (r\€(x)))— &x) = i. 

ax ax 

When x e R rf , then 

df{x) 



X f(x) = Yb\x) 1 - 

tt dx ' 

X l f(x ) = f jg -(x ) dfix) 
Then (*) is of the form 



,= i dxi 






. , 3x 



3 

to 
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= 2,2,— ir (I) - 

So the same results apply for x t e W l , but generally not for more 
than one "input", i.e., for vector w(i). 

Interpretation: x t defined by (J2ji makes sense for any w{.) e C(R+). 
In particular, if w(f) is a sample path of Brownian motion, then what 87 
equation does x, satisfy? 
Answer: the Stratonovich equation 

dx t = b(x t )dt + g(xi)odw t (3) 

Exercise 2.1. Expand x t given by (|2j using Ito's calculus and show that 
it satisfies @. 

The following examples show that the pathwise solution idea cannot 
generally be extended to "multi-input" equations. 



Example 2.1. Let 



x = g\x)w 1 +g 2 (x)w 2 



x(o) = X. 
The solution should be of form 



x t = h(wl,w 2 ). (4) 



and 



Then with h\{w l ,w 2 ) = — -h(w l ,w 2 ) etc., we have 
dw l 

X t = h\W + /l2W 



h\(w\,w 2 ) = g x oh(w),w 2 ') 
h 2 {w),w 2 ) = g 2 oh{w),w 2 ) 

h\ 2 {w],w]) = g\oh.h 2 = (g\oh)(g 2 oh) 
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h 2 i(w], W 2 t ) = (g 2 x oh)(g l oh). 



So we must have 



5 Sx 8 ax- 



88 Define the Lie bracket [Xi , X 2 ] = XiX 2 - X 2 Xi . Now 

dx \ dx 

1 2 r 1 2 r 

8 8 x fx + 8 8 fxx- 



Therefore 



[X l ,X 2 ]f = (g 1 g 2 x - g 2 g l x )fx. 



So a necessary condition for © to hold is that 

[X!,X 2 ]=0 
i.e., X 1 X 2 = X 2 X 1 . 

Exercise 2.2. Consider 

Let cp'(t, x) be the flow ofg' and t;)(x) = <p\w l t , x). Then show that 

if[X i ,XJ] = OVi,j. 

With one input, \\w n - w\\ — > implies x" — > x t , where || • || is the 
sup norm. But with inputs w l ,w 2 , the solution map generally is not 
continuous. 

89 Example 2.2 (Sussmann [23 ]). Let t e [0, 1] and 

x" = Ax n w hn + Bx"w 2 ' n 
x(o) = x 
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where A, B are n x n matrices with 

[A, B] =AB-BA±0. 



n n 



J = 1,2,..., n. 



w l <" 



Partition [0, 1] into n equal intervals I n 

Partition each I" into four equal intervals P. A = 1,2,3,4. Define 
to be equal to An 111 for t e I n 1 to -An 112 for I n 2 , and to zero for all other 
t. Similarly, let w 2 ' n be equal to 4n 1/2 for t e I n 2 to -4« 1/2 for t e P jA , 
and to zero for all other t. 

Then 

w i ' n (t)= f w Un {s)dsA- = 1,2. 



Clearly w l - n converges to zero uniformly as n — > oo, j = 1,2. Let 
^ = n~ 1/2 , then 



x"(l/«) = e as e- As e as e As x 



We use the Baker-Campbell -Hausdorff formula e A e B = e c where 
C = A + B+ l -[A,B] + -L{[[B,A],A] + [[B, A], B]} + ■■■ 



we get 



r = [B,A]-+o(l/n). 
n 



So 90 

x"(l) = e nT x (> 

_ e ([B,A]+o(l/n)) x 



Hence 



lim x" = g r[B ' A] x . 
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3 Pathwise Solution of the Filter Equation 

Consider the equation 

df(x t ) = Y f(x t )dt + Zf(x t )ody t + Yif{x t )odb l t . 

To express this pathwise in (y t ), let <f>(t, x) be the integral curve of Z 
and g t (x) = (f>(y(t), x). Define r\ t as follows: 

dfirit) = (^ U Y )f(rj t )dt + (^ l Yj)f(rj t ) o db[ 

n ( , = x. 

Then 

x t = gtOTJt 

= 0(y(O,/XO)- 
The generator of rj is 

j 
The Kallianpur-Striebel Formula: Recall 

<r,(f) = E^ 



= E (b) 



where D = Z + h. 



f(x t ) exp I h(x s )dy s - - I h 2 (x s )d; 
f{x t ) exp I h(x s )ody s - — I Dh{x s )ds 



Notation: For any diffeomorphism iff : M -> M, t//* : C°°(M) -> C°°(M) 
is given by 

«A7(x) = fom = Mix)). 

So 



/(*,) = dfirit), 



3. Pathwise Solution of the Filter Equation 



87 



and o- t (f) = E 



(//) 



£707r)exp| I £h(n s )ody s - - I Dh(n s )ds 



The next step is to remove "ody". Define 

H(t,x) = I (p* s h(x)ds. 

Jo 

Calculate H(y t , n t ) using Stratonovich calculus 

dH(y t , rj t ) = e,Krit)ody t + &Y Hy s (T] s )ds + ^Hy^od^. 

Notation: 

g s {x) = H(y(s),x) 

BJ(x) = 0*/(x)expl I 4>* u h(x)du 

Finally, we get 

o- t (f) = E^[By (t) f(n t )a' t '(y)] 
where the multiplicative functional a s t is given by 



a s t(y) = exp 



j Y*g u (n u )dbl -if (Yp 2 g u (7] u )du 

- I Y*g u (n u )du - - I f u Dh(n u )du 
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So cr f (/) is now pathwise in v with cr t (f) : C[o, £] — » R and (o~ t (f)/ 
cr, Q) is a version of £[/(x f )|y r ]. Now we want to compute o- t (f) re- 
cursively. 

(a) Multiplicative Functional Approach: 

Let (x t ) be a Markov process with extended generator (A,D(A)). 
The associated semigroup on B(E) is 



T Stt f(x) = E StX [f(x t )] 

a*(s < t) is a multiplicative functional (m.f.) of (x t ) if a\ is o~{x u , s < 
u < t) -measurable and for r < s < t, 



Corresponding to a\ there is a semigroup defined by 

T° t f(x) = E StX [f(x t )a s t ]. 

In particular, 

Tl t \ = E s , x [a s t l 

93 It is a Markov (or Sub-Markov) semigroup when 

E s , x [a s t ] = 1 (< 1). 

If (x f ) is a homogeneous Markov process, a\ is a homogeneous m.f. 
if 

a\ = a s t + r r o 6 r 
where 6 r x t = x t+r . 



Then 



a] = a"_ s o 6- s . 



So denoting a t = a", the m.f. property is 

a, +s = a t .a s o t . 

Now we want to find the generator of 

T? f(x) = E x (f(x t ) a t ). 

Suppose for the moment that a t < 1,V t. Then a t is monotone 
decreasing. In this cases a t corresponds to "killing" at rate (-da t /a t ). 
It is possible to construct an "a-subprocess" which is a Markov process 
x t such that 

E x [f(xf)] = Tf f{x). 
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See Blumenthal and Getoor [1]. Define the extended generator of 
T" to be the extended generator of xf , i.e., 

t 
f(x?)-f(x«)-J A a f(x° s )ds 

o 

is a local Martingale if / e D(A a ). This says (excluding stopping) 

t 
/«)-/(*?)- J A a f(x a u )du\F s 

s 

t 

or E Xi [a t - s f(x t )] - f(x s ) - E Xs I a u - s A a f(x u )du = 0. 

s 

So equivalently, / e D^") if 

t 
a, (f(x t ) - f(x) - J aiA tt f(x s )ds 



is a local Martingale (P x ) for every x. 

This characterizes A a even when the condition a t < 1 is not satis- 
fied, so we adopt it as our definition. 

Example 3.1. Let y, = exp I- J V(x s )ds\ where V e B(E). Take 
/ G D(A) and compute 

d{jt /(**)) = y t A f(x t ) dt + y t dM t f- V(x t ) f(x t ) y, dt. 
t t 

y t f{x t ) - f{x) = J y s [Af(x s ) - V(x s )f(x s )] ds + J y s dMJ. 
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So 



A^ f{x) = Af(x) - V(x)f(x). 
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Example 3.2. Let f5 t 



a(x t ) 
a(x ) 



rP 



where a e D(A) and a(x) > o V x. Then 95 



1 



rf g(x) = — T t (af)(x). 



Exercise 3.1. Show that 

A^fix) 



a{x) 



1 
a(x) 



A(af)(x). 



Now suppose x t satisfies 

df(x t ) = X f{x t )dt + Xj f{x t ) o dw\. 
Takeg e C™(E) and define 



f t 



S t = exp 



If we define 



■ I Xjg(x u )dw J u - - I Yjy x iSi x u)) 2 du 



dP, 



St, 



then 



dw\ = dwj + Xj g(x t )dt 



is a Pi - Brownian motion. Thus 



(1) 



df{x t ) = (X f(x t ) - J] Xjg(x t )Xjf(x t ))dt + Xjf(x t )o dw t j . 



Now S t is a m.f. of x f (as will be verified below) and 



So 



A 6 fix) 



E x [f(x,)S t ] = E d x [f(x t )l 



X -J j X jg (x t )X j f+l^XJf. 
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The three examples here are related by 



I Xjg{x s )dw{ 



g(x t ) - g(x) - Ag(x s )ds. 



I 



Using this in Q, we see that 8 t factors 

St = fit Jt 



with 



V(x) = -Ag(x) + \ Y, (Xjg(x)) 2 



-g{x) 



So 



So 



a(x) = e 



A 6 f(x) = eZA(e-tf) 



*8-^(Xjg) 2 



fix). 



eSA(eSf) = Af - ^(X jg )Xjf - Ag + \ ^(X jg ) 



f 



Exercise 3.2. Verify that this result is correct by direct calculation of 
eU(e- g f). 

We have the unnormalized solution of filtering problem as 

o- t (f) = E[B y , f(vt)a' t '(y)l 
for y e C[o, T]. Here n t is a diffusion, a°(y) is a m.f. of r] t . Now 

It t 

- J Y*g(n u )db J u - l - J Yj {Y *Mriu)fdu 
s s J 

x exp I § Yp]g u (j, u )?du - l - f(Y*) 2 g u (n u )d 
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T T 

- I Y*g u (7] u )du - - I Cu Dh< Jlu)du 



97 This factors af into product of a "Girsanov" m./. and a "Feynman- 

Kac" m./. Hence the corresponding generator is 



A y t f = A* t f-£r jgt r J f+ 



l -Y, { y>^- A *st-\e t Dh 



f. 



Proposition 3.1. A y J = B yt (A - \D 2 )b~}. 

Proof. This can be verified by a straightforward but somewhat lengthy 
calculation, using the expansion for e g Ae~ g obtained previously, once 
has obtained an expression for S" 1 . Recall that B t is defined by 

r 

B t f(x) =/(£(/, x))exp I h(£(u,x))du. 

o 
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It is a group of operators with generator D = Z + h. The inverse B t l 
is given as follows. Let g(x) = B t f(x), then 



-l. 



fix) = B; l g (x) 



I t 



= giSi-t, x)) exp 



fh({(u, 



£(-t, x)))du 



\ o 

( t 



g(£ \t,x))exp 



fh(C\s, 



x))ds 



Example 3.3 (Independent "signal" and "noise"). Take Z = 0, then 
g(t, x) = x and 

A] f{x) = JVM [a - l -hA(e-y^f(.))(x) 
= e^Ae-^f - -h 2 f. 
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It is easy to see that this must be the right formula. The calculations 
have been carried out for arbitrary y e C[o, T] but A y t depends only on 
y(t). So A y = A y where y(s) = y(t) (t fixed). Now 



<r,(f)(y) = E 



= E 



f(x t )e h{x ' )y > exp 



f(x t )e h{x < )y > exp 



- I y{s)dh{x s ) - - I h 2 {x s )ds 



t 

y{t)h(x s ) + y(t)h(x ) - - J 



V 

( 



h (x s )ds 



f( Xi ) c hMy> ex v ( - y(t)h(Xt)) 

exp(-y(t)h(x )) 



.exp 



~\^ 



(x s )ds 



So we have separated into two functionals and the result follows. 

Direct Solution of Zakai Equation: We will consider a slight general- 
ization from the original Zakai equation. Define 






+ Y + h 



D =Z + h 



where Fj, Z are smooth vector fields; h, h are C£° functions (Previously, 99 

1 r 

we had h = — Dh). Write (f,fi) for J f d^ and consider the measure- 
valued equation 



d(f, o- t ) - (Lf, cr t )dt + (Df, cr) o dy t 



(2) 



where </, cr.) = f(x) i.e., cr = 6 X . The solution can be expressed as 
follows: Define x t by 

df(x t ) = Zf(x t ).dy, + Y f(x t )dt + Yj(x t ) ° db J t ,x (> = x, 

where bj are Brownian motion independent of y. Then the solution is 

)exp I h (x s )ds + I h(x s )ody s 



o-t(f) = E b 



f(x t )i 
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Kunita fTTl show that this solution is unique if coefficients are 
smooth and bounded. Now the question is whether cr t has a density. 



Theorem 3.1. 

where 



(f,CT,) = (B y J,V t ) 



(3) 



By, fM = (f,H,) 

and fi t , v t satisfy the equations 

d < f,Ht> =< Df,/j, t > ody t 
<f,m > = /(*) 

d < f v, > =< B yi L B~j,v t > dt. 

100 Proof. If L = o, then © is the same as ©; so the solution of © is 



(4) 
(5) 



/ t 



<f,l*t >=/(**) exp 



I h(x s ) o dy s 



where x t satisfies 



df(x t ) = Zf(x t )ody t . 



But this has pathwise solution x t = g(y t , x). The previous definition 
of B was 



B y ,f(x) = M(y t ,x))exp 



I h(£(u,x))du 



Now, 



l yt 



I h(g(u,x))du 



h(g(yt, x))ody t 



So © holds with B, defined as before. Now 

d < B y J, v t > = d! < By f f, v t > + < B y J, v t > dt 
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= B y ,L Bj t B y J, v t > dt+ < B yt D f, v t > ody t 

by and ©. 
=< By t Lf, v t > dt+ < B yi D f,v t > ody t . 

This verifies (J2ji. 

Proposition 3.2. Suppose v t has a density function q t (z, x) |j Then for 101 
t > o.cr t has density 

t y> 



p t (.V) = q t (r\yr,V)x)exp 



f h(rHs,v))d S 



xi— rV^oi (6) 

oV 



dC x 1 

where | | is the Jacobian of the map V — > £, (y t , V). 

dV 

Proof. If v t has a density q t , then 

< f,o- t > = I B yi f{z)q t {z,x)dz 

yt 
= I M(yt,z))exp( I h(£(u, z))du)q t (z, x)dz. 

o 

Changing the variable to V = g(y t , z) gives (J6J. n 

Theorem 3.2 (Bismut [2]). v t has C* '-density if the Y t are "smooth" 
vector fields, i.e., coefficients are bounded with bounded derivatives of 
all orders and Y\,...,Y„ satisfy the "restricted Hormander condition" 
H :- Consider vector fields 7,-, [7,-, Yj\, [[Y,-, Yj], Y^] .... At each x the 
restrictions of these vector fields to x span T X (M). 

In local coordinates 7 r = £ b t (x) — , . . . etc. So the condition says 

i dx { 

the vectors b etc. span R d at each x. Recall, B }l LB~j = A* + (1 st and 



3 Here z is the "dummy variable" and x refers of the initial condition in J2j, i.e., 
0"o = $x 



2 + 
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th order terms) 

Now 

&[Y i ,Y j ] = [&Y i ,{- 1 Yj] etc. 

102 So if Yi satisfy the Hormander condition, then (£"' Yj) satisfies it. 
Hormander's own result requires coefficients to be C°° in (t, x). Here 

the coefhcients are continuous (but not even C 1 ) in t. Bismut's version 
of Malliavin calculus shows that the result still holds with this degree of 
smoothness in t. 

In the filtering problem, the "signal process" involved vector fields 
X\,X2, . ..X h ,Xq and Y = AX, where A is nonsingular if | a \< 1. Then 
X = A~ l Y. So 

[. . . [[X h , X h ]X h ] . . . X ik ] = 2 cj [. . . [[Ytf, Y 2 6], Y 3 5] . . . Y inJ d] . 

j 

So if the "X" Lie brackets span W l then there must be a collection of 
"7" brackets which also span R d . The Hormander condition for X with 
| a \< 1 implies the existence of density. 

The Case of Vector Observations: Let df = h t {x t )dt + dW° J , W 0J are 
independent Brownian motions, a will now be a matrix, 

atjt =< W QJ , W 0J > . 

Consider the following cases, (a) Independent signal and noise: 
Here ay = 0V/, j. Then whole theory goes through unchanged. 

103 Then (J2ji becomes 



with solution 



d < f,[i t >=2_ u < h f, fi t > ody l t 



<f,Ht> = exp ]Y y t (t)h(x) f(x) 
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= Y\exp(yi(t)h(x))f(x) 
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So this gives pathwise solution as before. 
Another Point of View: The Kallianpur-Striebel formula is 



cr t (f) = E 



(.*) 



( t 



f(x t ) exp 



r« 



2 /' 



s)ds 



f(.x t )Y\J w "exp 



J(t)Kxt) 



(s)dh((x s ) 



( t 



I / *?(*. 



)ds 



(b) The General Case: Here we have no "pathwise" theory (except 
under very artificial conditions) but the same theory goes through a.s. 
(Wiener measure). There is no continuous extension to the whole of 
C p [o, T]. In this case, equation (J2ji becomes 



where 



d < f,n, >= ^ < Djf,Ht > ody\ 

i 

Dj = Zi + h and Z, is a vector field 



A pathwise solution only exists if D'-s commute, which is very arti- 
ficial. But, as before, the solution can be expressed as 



t t 



< f,/u t >= f(x t )exp 
where x, satisfies 



^ I hj(x s )ody' s 



V ' o 



(7) 
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df(x t ) = 2j Dif{x t )ody\ 



98 






Regard fj. as the operator mapping / — »< f,/j t >. Then "stochastic 
flow" theory (Elworthy U3, Kunita El, Bismut says that if D^s 
have smooth coefficients then x — > x t (x,a>) is a diffeomorphism a.a. a», 
and so the inverse map xj (x) exists. We have to calculate fij . Gener- 
alize Q slightly to 



<f,fx s ,t >= f(x„t)exp 



2_j I hj(x r ,r)ody' r 



df(x t , t) = Yu A-/C**, t)ody\t>s 



Proposition 3.3 (Kunita fill ). 



j"vk/W) = /(*J(>))exp 



-J^fhiix-jix^odyl 



(8) 



where "od" means backwards Stratonovich integral. Here define o~— 
fields F r>t r < t, by 

F r,t = cr{y i u - y[, r<u,v<t,i = l,2,...,d). 

t 
Then J <p r dy l is a well defined backward I to integral if <p r is a back- 

s 

105 ward semimartingale w.r.t. (F r j) r < t . Then the Stratonovich integral is 
defined as usual. If® r is continuous, then 

So iu~](x) is well defined by ©. Now verify that 

t 

Jhi{^(P))°U\ p=Xst(x) 



3. Pathwise Solution of the Filter Equation 



t 



I 



s 



hi{x s>r )ody l r . 



This checks that ji s I(ji s j(x)) = x. 



Now all remaining calculations go through as before but only a.s. 
(Wiener measure). 

More general results on existence of densities have been obtained 
by Bismut and Michel [ 3 ] 
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